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INTRODUCTION 


Despite  significant  study  and  corrective  effort  over  a  period  of  two 
decades,  the  defense  system  acquisition  process  in  the  U  S  continues  to 
be  plagued  with  major  cost  overruns,  schedule  slippages,  and  hardware 
performance  deficiencies.  The  paradox  is  that  most  people  in  government, 
the  military  services,  and  industry  who  work  in  this  area  honestly  desire 
that  this  situation  be  otherwise— and  they  share  a  growing  personal  frus¬ 
tration  that  improved  results  have  not  been  produced  in  spite  of  redoubling 
of  their  individual  efforts.  By  implication,  there  is  something  very 
fundamentally  wrong  or  lacking  in  our  policies  and/or  practices  which  has 
escaped  identification  and  attention. 

At  the  outset  of  his  tenure,  the  present  Deputy  Secretary  of  Defense 
focused  on  cost  growth  in  systems  acquisition  as  a  problem  which  should 
be  given  priority  consideration.  Optimism  ia  program  cost  estimates, 
cost  growth  via  excessive  development  and  production  changes,  failure  to 
identify  major  risk  areas,  and  excessive  dependence  on  paper  analyses 
were  identified  as  the  principal  causes.  In  his  memoranda  of  31  July 
1969  and  28  May  1970,  the  Deputy  Secretary  directed  the  Service  Secretaries 
to  identify  areas  of  high  technical  risk,  to  accomplish  "formal  risk 
analysis,"  and  to  expand  program  management  practices  to  include  explicit 
consideration  of  risk  assessment,  risk  reduction,  and  risk  avoidance. 

The  application  of  risk  analysis  as  defined  in  this  report  appears 
to  have  great  potential  at  all  levels  of  the  acquisition  process — from 
the  program  manager  to  the  OSD  pclicy  guidance  staffs.  However,  in  its 
present  state  of  evolution,  risk  analysis  is  not  a  science — or  even  an 
art.  Indeed,  in  the  context  of  the  systems  acquisition  process,  it  is 


Chapter  V  addresres,  in  a  normative  fashion,  the  subject  of  how  to 
perform  a  risk  analysis,  what  the  outputs 'should  be,  and  some  of  the  uses 
and  benefits. 

Chapter  VI  presents  the  conclusions  and  recotnoendations  of  this  study 

team. . 

The  authors  are  indebted  to  more  people  in  the  academic  community,  BOD, 
and,  industry  than  we  can  acknowledge  individually.  Without  their  cooperation, 
guidance,  and  experience-together  with  generous  contributions  in  time, 
travel,  and  creative  suggestions— the  study  team  could  not  have  penetrated 
very  far  into  the  intangible  subject  of  risk  analysis.  Of  utmost  importance 
was  the  generous  support  of  Edward  Ball  and  James  Grodsky,  Research  and 
Development  Policy,  ODDR&E,  and  the  management  research  grant  provided  by 
the  Aeronautical  Systems  Division,  AF  Systems  Command.  Particular  recog¬ 
nition  mo^t  also  be  made  of  the  spontaneous  responses  by  Army  and  Navy 
Department  personnel  at  every  level  which,  together  with  Air  Force 
contributions,  gave  the  researchers  a  balatnced,  tri-Service  perspective 
concerning  on-going  programs  and  risk  analysis  activities .  Finally,  it 
is  importaint  to  recognize  that  this  report  represents  the  views,  con¬ 
clusions,  and  recommendations  of  the  study  team  and  has  not  been  officially 
approved  by  any  DOD  staff  or  line  agency. 
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EXECUTIVE  SUMMARY 


Introduction 

€. 

,'>This  study  investigates  a  method  .of  reducing  cost  growth  and  improving 
quality  in  the  weapon  system  acquisition  process.  Other  recent  analyses 
have  focused  on  identifying  and  correcting  major  deficiencies  in  current 
policy  guidance  and  management  procedures.  This  study,  by  an  ir.terdis- 
v  ciplinary  team  of  Air  Force  Academy  faculty  members *  takes  a  mere  specu- 
'lative-approach  by  investigating  a  new  management  process,  "formal  ribK 
analysis."  ■  \ 

/  Risk  analysis  is  presently ‘neither  an  art  nor  a  science.  As  a 
result,  the  informal,  fragmented  efforts  attempted  in  the  past  have  not 
been  effective  in  solving  the  overall  /problem  whidh  is:  so  complex ,  so 
ill  defined  and  nebulous  that  arriving  at  acceptable  definitions, 
discovering  the  major  sources  of  risk,,  developing  working  models  for 
effective  analysis  and  evaluation,  and  distilling  pertinent  information 
and  techniques  into  meaningful  guidelines  are  themselves  problems  of 
staggering  magnitude. 

^Different  organizations  use  the -terminology  of  risk  analysis  in 
different  ways;  therefore,  it  is.  necessary  to  clarify  risk  analysis  as 
used  in-- this  report?^ Risk  is  the  probability  that  a  project  will  not 
be  completed  within  specified  time,  cost  and  performance  constraint's  by 
following  a.  specified  course  of  action.  Risk  assessment  is  an  estimate 
of  the  risk  associated  with,  a  particular  course  of  action.  Risk  manage¬ 
ment  is  the  generation  of  alternative  coursed  of  action  for  reducing 
risk.  Risk -analysis  is  the  larger  process  of.  combining  risk  assessment 
and  risk  management  in  order  to^examine. factors  affecting  the.  risk  of 
acquiring  a  system.  It  is  the,  purpose  herein;  to  identify  what  a  risk 
analysis'  is,  how  it  can  be  accomplished,  who  should  accomplish  it  and 
where  it  fits  in  the  management  structure  for  weapons  systems 
acquisition;. >  \ 


Risk  and  Uncertainty 


To  determine  the  risk  associated  with  the  acquisition  of  a  weapon 
system,  it  is  necessary  to  do  the  following!  (-1)  Establish  the  dominant 
uncertainties  which  are  present;  (2)  Select  a  promising  course  of  action 
based  on  these  uncertainties;  (3). Assess  the  risk  of  this  course  of 
action;  (4)  Generate  alternatives;  (5)  Assess  their  risks,  and  (6) 
Iterate  to  a  point  of  diminishing  returns.  This  is  a  risk  analysis, 
and  it  can  be  used  as  one  input  in  the  selection  of  a  preferred  course 
of  action. 


The  study  team  found  that  uncertainty  in  the  weapons  system 
acquisition  process  fell  naturally  into  four  relatively  clear-cut 
categories;  TARGET,  TECHNICAL,  INTERNAL  PROGRAM  and  PROCESS.  Target, 
uncertainty  involves  a  lack  of  knowledge  concerning  what  end  items  are 
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desired  and  what  criteria  should-  he  used  to  evaluate  them.  Technical 
uncertainty  involves  solving  technical  problems.  Internal  Program 
uncertainty  involves  uncertdihtjr  about  how  "a  program- should- -be  planned 
and  managed.  Process  uncertainty  involves  the  program’s  interaction 
with  the  external  environment-  and  revolves  around  uncertainty  over  the 
availability  of  the  resources  required  to  complete  a  program"  and  the 
criteria  Snd,  thresholds,  eiaployed  in  program  (approval.  Experience  has 
shown- that  this.  Categorizing  sharpens  analysis  of.  how  uncertainty  acts 
to  generate  risk. 

One  result  of  this  study  is  the  discovery  that  many  causes  of  cost 
growth  are  introduced  early-  in  a  program  before- the  Program  Manager 
assumes  responsibility.  Thus  an  important  task,  for  the'  study  team  has 
been  to  identify  and  categorize  the  major  uncertainties  inherent  in  the 
program  and -outside  the.. jurisdiction  of  the- program  Manager. 


Bisk  management  involves  the  selection  of  a  set  of  alternative 
-Courses  of  action  to  reduce  or  avoid  risk.  Initial  alternatives  are 
selected  based  on  the  categories  of.  uncertainty  that  dominate  the  pro¬ 
gram  arid  differ  according^  to  the  type,  of  uncertainty.  For  example,  if 
target  uncertainty  is  predominant  :  (l)  threat  studies  -need,  to  be  con¬ 

tinued  through  the  development  period;  (2)  parallel  development  of 
competing  systems  is  not  appropriate;  (-3)  .performance  requirements  heed 
-to  be  stated  in-  -terms'  of  intervals  rather  than  points;  (4)-  a  source  -for 
tradeoff  decisions  between  the.  Program  Manager  and  OSD  authorities  is 
desirable;  (5)  a  restatement  pf ^performance  requirements  is  needed  after 
development  source  selection;  .and  (6)  ipperational. prototyping  should  be. 
considered. 


If  technical  uncertainty  is  (paramount:  (l)  model  testing  and 
development,  production,  or  operational  prototyping  needs  to-be  con¬ 
sidered  in  lieu  of -paper  analyses;  (2)  parallel  development  by  more 
than  one  approach  is  indicated  for  quantum  technology  jumps;  and  (3) 
subcontractors  with  a  proprietary  state-of-the-art  advantage  should  be 
made  available  to  all  prospective  contractors. 

Internal  program  uncertainty  requires:  (l)  a  maximum  of  flexibility 
in  program  design;  (2)  increased  communication, among  Program  Managers;; 
(3)  high  visibility  for  military  Program  Managers  to  improve  the  career 
aspects  of  their  jobs;  and  (4)  possible  production  prototyping. 

Process  uncertainty  encompases  a  multitude  of  unknowns  and  indi¬ 
cates:  (l)  the  need  for  program  approval  to  be  clearly  defined;  (2) 
that  tradeoff  authority  that  is  beyond  the  Program  Manager  bp  rapidly 
available  to  him;  and  (3)  the  Program  Manager  should  conduct  a  con¬ 
tinuing  sensitivity  analysis  of  possible  impacts  due  to  unexpected 
funding  changes. 


5 


I 


Risk  Assessment 


A  risk. assessment  isa  comprehensive,  carefully  structured  approach 
for  estimating  the  risk  profile  of  a  particular  alternative  course  of 
action.  A  risk  assessment  of  eaehof  several  alternative  courses  of 
action  is  then  part  of  a  risk  analysis. 

A  Quality  risk.*assessment  can  be  accomplished  only  by  &  group  of 
trained  analysts  in  mathematics,  probability,  statistics,  operations 
research,  and  computer  science  -- -aided  by  cost  analysts,  production, 
design,  and  engineering  people,  and  experts  in,  various-  technical 
disciplines. 

The  quantitative  disciplines  involved  in  risk  assessment-  are  group 
assessment  techniques,  subjective  probability,  trend  extrapolation, 
cost  estimating,  end  network  analysis.  The  first  four  can  be  used  to 
produce-  terminal  results  or  to  provide  inputs  to  the  more  powerful 
technique  of  network  analysis-;  The  outputs  which  they  produce,  though 
useful,  do  riot  supply  the  kind  of  information  necessary  for  a  quanti- 
tativw  risk  assessment,  of  the  type  required'.  The  technique  which  offers 
the  most  promise  in  quantitative  risk  assessment  is  a  versatile,,  simu¬ 
lated  network  approach  using  group,  assessment  techniques,  subjective 
probability;  technological  forecasting,  cost  estimating,  arid5  others  as 
sources  of  input. 


Risk  Analysis 

The  techniques  for  conducting  a  risk  analysis  exist;  however,  a 
single  procedure  should  hot  be  established  since  the  best  procedures 
will  differ  with  various  projects,:  On  the  other  hand,  the  outputs  of 
an  analysis  cari  be  fairly  well  specified. 

1.  A  general  description  of  the  dominant  uncertainties  (target, 
technical,:  internal  program,  or  process:)  which  directed  the  selection 
of  the  original  course  of  action. 

2.  Identification  of  alternate  courses  of  action  (such  as 
hardware  proofing,  parallel  development,  etc.)  to  resolve  the  major 
uncertainties. 

3<  A  detailed  discussion  of  the  potential  problems  in  each 
major  program  element  for  each  course  of  action  considered. 

4.  Individual  .and-  joint  risk  profiles  of  time  and  cost  for 
each  alternative-  course1  of  action.  The  inherent  assumption  is  that 
the  specific  desired:  performance  is  obtained  by  following  the  course 
of  action. 

5.  An  arialysis  of  how  sensitive  the  risk  profiles  are  to 
change  in  the  inputs. 
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6.  Tradeoffs,  as  recommended  by  the  Program  Manager,  for 
maintaining  the  overall  program  within  specified  cost,  time,  and 
performance  thresholds-. 

7.  Comparison  with  previous  risk,  analyses  of  the  same  project. 

8.  A  comparison  of  the  candidate  management  courses  of  action 
and  a  recommendation  of  a  preferred  course  of  action  on  the  basis  of 
risk  considerations  alone. 

9.  A  discussion  of  the  major  assumptions  and  an  explanation  of 
the  disparity  when,  the  results  are  different  from  those  expected. 

The  concept  that  a  risk  analysis  requires  an  iterative  interaction 
between  the  manager  and  the  assessors,  coupled  with  the  fact  that  oaly 
the  contractor  has  ready  access  to  the  vast  amount  of  necessary  data, 
dictates  that  the  analysis-  must  be  performed  in  the  military  or  con¬ 
tractor  Program- Office , 

The  contractor  should  continually  assess  the  total  program  risks. 
The  prime,  benefactor  of  a  risk  analysis ,  the  Program  Manager,  can. 
then  request  an  analysis  of  several  alternative  courses  of  action  each, 
•time  the  assessment  indicates  the  need  for  a  tradeoff  decision  in  a 
.program  element .  Although  a  risk,  analysis  would  he  hut  one  input  t-o 
such  a  decision,  it  could  be  an  important  one , 

The  results-  of  a  thorough  risk  analysis  should  be  presented  by  the 
Program  Manager  each  time  the  program  is  reviewed  for  a  major  decision 
by  high  level  Service  authority.  Such  :an  analysis  ordinarily  includes 
■a.  comprehensive  study  of  several  .alternative  courses  of  action  for  con¬ 
sideration  by  the  decision-  maker.  For  a  DSARC  or  congressional  review, 
the  risk  profiles  for  time  aha  cost  for  the  preferred  course  of  action 
should  be  presented.  These  profile#*  -provide  immediate  identification 
of  the  program  risks  fixed  by  the  DCP  thresholds  and  serve  as  useful 
inputs  to  adjustment  of  those  thresholds.  The  reviewing  authority 
cannot  conduct  a  meaningful  independent  risk  analysis.  For  further 
objectivity;  however ,  .he  could  request  an  independent  evaluation  of 
the  Program  Manager's  analysis. 


Conclusions  and  Recommendations 


1.  Conclusion; 

ONE  OF  THE  BASIC  PROBLEMS  IN  ANY  STUDY  ON  RISK  IS  THE  LACK  OF 
A  GENERALLY  ACCEPTED  GROUP. OF  DEFINITIONS. 


Recommendation; 

ESTABLISH  DOD  DEFINITIONS  OF  THE  BASIC  TERMS  ARB  CONCEPTS  USED 
IN  RISK  ANALYSIS; 
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RISK:  The  probability  that  a  planned  event  will  not 
be  attained  within  constraints  (cost,  schedule, 
performance)  by  following  a  specified  course  of 
action. 

UNCERTAINTY:  incomplete  knowledge. 

RISK  ASSESSMENT:  A . comprehensive  arid  structured 
process  for  estimating  the  risk 
associated  with  a  particular  alter¬ 
native  course  of  action;  also  the 
product  of  such  a  process. 

RISK  MANAGEMENT:  The  generation  of  alternative  courses 
of  action  for  reducing  risk. 

RISK  ANALYSIS :  The  process  of  combining  the  risk 

assessment  with  risk  management  in  an 
iterative  .cycle;  also  the  product  of 
such  a  process. 


.2.  Conclusion: 

TECHNICAL  UNCERTAINTY  MAY  BE  ONLY  A  SMALL  PART  OF  THE  WEAPON 
SYSTEM, ACQUISITION  PROBLEM,  SUCCESSFUL  PROGRAM  DIRECTION  ALSO 
REQUIRES  MANAGEMENT  CONSIDERATION  OF  TARGET,  INTERNAL  PROGRAM, 
AND  PROCESS -UNCERTAINTIES . 

Recommendation : 

REQUIRE-  ANY  "RISK  ANALYSIS"  TO  INCLUDE  CONSIDERATION  OF  TARGET, 
TECHNICAL,  INTERNAL  PROGRAM,  AND  PROCESS  UNCERTAINTIES , 


3.  Conclusion: 

IN  THE  AREA  OF  QUANTITATIVE  RISK  ASSESSMENT,  AGGREGATION 
TECHNIQUES  (SUCH  AS  NETWORK  ANALYSIS)  ARE  FAR  MORE  ADVANCED 
THAN  THE  TECHNIQUES  FOR  OBTAINING  INPUT  DATA  (SUCH  AS  SUB¬ 
JECTIVE  PROBABILITY  AND  TECHNOLOGICAL  FORECASTING) . 

Recommendation : 

FUNDING  PRIORITY  FOR  IMPROVING  METHODS  FOR  QUANTITATIVE  RISK 
ASSESSMENT  SHOULD  BE  GIVEN  TO  DEVELOPMENT  OF  INPUT  TECHNIQUES. 


4,  Conclusion:  NOT  .REPRODUCIBLE 

RISK  ANALYSIS-  'PRIMARILY  BENEFITS  THE  PROGRAM  MANAGER ,  NO 
AGENCY  OUTSIDE  THE  MILITARY  OR  CONTRACTOR  PROGRAM  OFFICE  CAN 
EFFECTIVELY  PERFORM  A  RISK  ANALYSIS  OF  THAT  PROGRAM, 
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4.  Recommendation : 

o  Direct  each  concept  formulation  contractor  to  perform  a  risk 
analysis. 

o  Direct  each  contractor  in  the  source  selection,  competition 
to  perform  a  risk  analysis-  and  specifically  include  at  least 
the  uncertainties  identified  by  the  risk  analysis  performed 
during-  concept  formulation. 

o  Require  the  Program  Manager  and  the  winning  contractor  to 
update  the  risk  analysis  after  source  selection  and  before 
contract  award, 

o  Direct  the  contractor  for  each  major  oii-going  program  to 
conduct  a  continuing  assessment  of  the  riBk  in  the  program. 

o  Require  the  Program  Manager  to  present  the  results  of  a 
current  risk  analysis,  to  include  alternative  courses  of 
action,  each  time  the  program  is  reviewed  by  higher  service 
authority  for -a  major  tradeoff  decision. 

o  Require  the  Program  Manager  to  present  the  results  of  a  risk 
assessment  of  his  program,  specifically  the  risk  profiles 
for  time,  and  cost,  each  time  the  program  is  reviewed  by  the 
DSARC.,  T  '  ' 

o  Require  an  independent  evaluation  of  the  risk  suialysis  or 
assessment  each  time  the  program  status  is  presented  to. 
higher  authority  for  a  major  tradeoff  or  milestone  review. 


5.  Conclusion: 

INITIAL  COST  AND  SCHEDULE  ESTIMATES  FOR  MAJOR  PROGRAMS  HAVE 
INVARIABLY  BEEN:  OVER-OPTIMISTIC .  THE  RISK  THAT  COST  AND  SCHEDULE 
CONSTRAINTS  WILL  NOT  BE  MET  CANNOT  BE  DETERMINED  Is1  COST  AND 
SCHEDULE  ESTIMATES  ARE  GIVEN  IN  TERMS  OF  SINGLE  POINTS  RATHER 
’■  THAN  DISTRIBUTIONS. 

Recommendation : 

REPLACE  THE  POINT  ESTIMATES  OF  COST  AND  SCHEDULE  PRESENTLY  USED 
WITH  THE  JOINT  RISK  PROFILE  FOR  COST  AND  SCHEDULE  FROM  THE  RISK 
ANALYSIS  DEVELOPED  AT  THE. COMPLETION  OF  SOURCE  SELECTION. 

6.  Conclusion: 

TO  OUR  KNOWLEDGE  NO  MAJOR  DOD  PROGRAM  HAS  DEVELOPED  OR  USED  A 
RISK  ANALYSIS ,0F  THE  MAGNITUDE  ENVISIONED  IN  THIS  REPORT. 


6..  Recommendation: 


INITIATE  TEST  CASES.  IMffiDIATELY.  FORMAL  RISK  ASSESSMENT  AND 
ANALYSES  SHOULD  RE  USED  THROUGHOUT  THESE  PILOT  PROGRAMS  TO 
DETERMINE  THEIR  FEASIBILITY  AND  UTILITY  TO  A  DECISION  MAKER. 


7-  Conclusion: 

THERE.  TS-  NO  "ONE  BEST  WAY"  TO  CONDUCT  A  RISK  ANALYSIS. 
Recommendation : 

THE  FOLLOWING  LIST  OF  OUTPUTS  FOR  A.  RISK.  ANALYSIS  IS 
RECOMMENDED.  PILOT  PROGRAMS  SHOULD  RSCC^ND.  THE  MOST  USEFUL 
OUTPUTS.  ...  . 


1.  A  general  description  of  the  types  of  uncertainties 
in  the  program. 

2.  A  detailed  discussion  of  the  potential  problems  in 
each  major  program  element  (engine,  etc.). 

3-  Identification  of  alternative  management  courses  of 
action  to  resolve  the  major  uncertainties. 

4.  Probability  distributions  of  time  and  cost;  risk 
profiles  for  each  course  of  action. 

5.  A  sensitivity  analysis  to  determine  the  effect  of 
input  perturbations  on  the  risk  assessment  output. 

6.  Tradeoff  studies  as  directed  by  the  Program  Manager 
for  maintaining  the  overall  program  within  specified 
cost,  time  and  performance  thresholds. 

7.  Comparison  with  previous  risk  analyses  to  identify 
trends. 

8.  A  comparison  of  the  candidate  courses  of  action  and  a 
recommendation  of  a  preferred  strategy  based  on  risk 
considerations  alone. 

9.  A  discussion  of  the  major  assumptions  and  an  explanation 
of  the  disparity  when  the  results  are  different  from 
these  expected. 


8.  Conclusion: 


THE  CONCEPTS  OF  RISK  ANALYSIS  ARE  INADEQUATELY  UNDERSTOOD.  AN 
EDUCATION  PROGRAM  IS  NEEDED  TO  INSTRUCT  ANALYSTS  AND  MANAGERS 
IN  THE  PREPARATION  AND  USE  OF  A  FORMAL  RISK  ANALYSIS. 
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8.  Recommendation: 


FOLLOWING  A  MEANINGFUL  PILOT  /PROGRAM,  DESIGN  AND  IMPLEMENT  A 
RISK  ANALYSIS  EDUCATION  FROGRAM-VlTH  THE  FOLLOWING  MAJOR  AREAS 
OF  EMPHASIS;  - 

Short  orientation  courses  in  risk  analysis  for  high  level 
officials  who  deal  with  uncertainties  in  program  management 
and  program-approval. 

Longer  training  courses  to  outline  the  details  of  risk 
assessment  techniques  for  selected  personnel  vho  may  be 
required  to  perform  or  evaluate  such  assessments  in  the 
government  and  industry  Program  Offices. 

Introduction- of  Risk  Analysis  as  a  discrete  subject  in 


CHAPTER  I :  THE  RISK  ANALYSIS  PROBLEM 


The-  Setting  of  the  Problem 

Cost  overruns,  schedule  slippage  and  performance  degradation  in 
American  weapons  acquisition .havereached  intolerable  proportions  at  all 
levels  of  American  society  and  government  .  Tfee^prevailing  attitude  is  one 
sharpened  by  the  fact  that  the  defense  of  the  nation  is  involved-,  and  the 
cost  of  weapons  is  going  up  as  the  ability  to  purchase  is  going  down.  There 
fire  also  severe  and  competing  social  -problems .  Equally  disturbing  is,  the 
fact  that  no  one  seems  ,aible  to  put  his  finger  squarely  on- what  is  wrong. 
Guesses  range  from  outright  fiaud  arid-conspiracy  ;oni  the  part  of.  .the  "military- 
industrial"  complex  to  "buiitrinHi;nef ficience s "  of  the  most  complex  weapons 


acquisition  . process  the  world 


seen. 


This  study  is  not  the  first  "hard  look"  at,  the  problem.  Several  note¬ 
worthy  groups  have,  produced  a  number  of  significant  studies.  At-  the  same 
time,  an  iupressive  huniber  of  individuals  have  developed  approaches  and 
techniques  for.  dealing  with  particular  aspects  of  the  situation.  This  study 
reflects  the  concern  at  the  policy  level.  It  is  thus  an  attempt  to  lock  at 
the  problem  as  .a  whole,  to  see  as  clearly  as  possible  both  the  magnitude  of 
the  problem,  and  possible  approaches  to  its  solution. 

The  solution  to  the  problem  obviously  depends  on  the  diagnosis  of  its 
cause.  If,  for  example ,  fraud  is  involved,  firm  and  enforceable  laws  may 
be  the  answer;  if  inefficiency,  better  management.  Unfortunately,  the 
problem  has  defied  simple  diagnosis  and  hence  simple  solutions.  Indeed, 
the  fact  that  so  much  of  the  cost  overruns  and  schedule  slippages  have 
more  or  less  caught  everyone  by  surprise  indicates  that  the  fault,  in 
large  part,  rests  with  factors  which  were  not  anticipated  when  the 


original  estimates  were  made.  This  alone  suggests' that  the  solution 
lies  in  better  anticipation,  assessment  and  management  of  these  factors, 
a  process  that  is  loosely  called  ’’Risk  Analysis." 

Study  uenesis  and  Approach 

This  study  was  undertaken  because  of  wide  interest  within  DOD.  arid — 

industry  in  the  requirement  for  "formal  risk  analysis"  which  the -Deputy- .... 

Secretary  of  Defense  directed  in  his  memoranda  of  3,1  July  1969  and  28  May- 

1970  (1,2)  .*  Additional  motivation  was  provided  by  OSD  policy  guidance: 

All  hew  programs  will  be  kept  in  the  conceptual  development- 
stages  until  the  responsible  Service  Secretary  and  the  OSD  can 
be  assured  that  the  program  is  actually  in  the  proper  shape  to 
proceed  into  full-scale  development.  (2,,,p>  3) 

The  objective  of  the  research  was  to  distill  available  expertise  oh  risk 
analysis,  to  examine  the ^benefits  and  utility  of  such,  an -analysis, 
to  suggest  possibilities  for  using  risk  analysis  ih  the  systems  acquisi¬ 
tion  process.  At  the  very  least  it  was  anticipated  that  the.  study 
would  reveal  what  kind  of  an  analysis  might  be  acceptable  to  OSD. 

The  study  team  established  contacts  with  management  consultants  and 
academicians,  with  program  managers  in  .the  Services,  and  industry,  and 
with  professional  groups  and  DOD  policy  staffs.  Through  these  contacts 
an  extensive  library  was  assembled  on  the  subjects  of  risk,  uncertainty, 
decision  theory,  and  the  defense  systems  acquisition  process. 

Status  of  the  Art 

Risk  Analysis  and  cost  analysis  are  sometimes  thought  to  be  comparable 
disciplines.  But  there  are  fundamental  differences  between  the  two. 

^Numbers  in  parentheses  refer  to  items  in  the  Bibliography  at 
the  end  of  this  volume. 
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Cost,  of  course,  is  an  explicit  parameter  measured  in  familiar  terms 
with  fairly  .simple  quantitative  techniques  for- data,  manipulation.  For 
example ,  derivation  of  total  program  cost— given  those  of  its  elements — 
is  not  a  calculation-  of  great  difficulty. 

Bisk,  on  the  other  hand,  is  quantified  in  the-  less  familiar  terms  of 
probability  and.  the  derivation  of  total  program  risk-  is  a  much  more 
difficult  process,  although  methods  So  exist.  Like  cost  analysis  risk 
analysis  is  best  performed  by  specialists.  The  major  .problem  common  to 
both  is  the .acquisition  of  valid  input  data. 

In  summary,  it  is  inifprtmiateiy  trua  that  risk  analysis— in  the  con¬ 
text  of  the  weapon  system  acquisition  process— does;  not  enjoy,  the. 
recognition-.and  use  as  does,  for  example,  cost  analysis.  Rather,  it 
is  at  best  an  art  in  a,  relatively  primitive  state  -of  development,  albeit 
one -with  the  gleam  of  considerable  promise , 

Areas  of  Application  for  Risk  Analysis 

Some  classical  studies  of  cost  growth  over  the  past  two 'decades  were 
accomplished  by  Peck  and  Scherer  (3)  and  Marshall  and  Heckling  (4). 

These  showed  the  average  USAF  programs  overran  original  cost  estimates 
by  a  factor  of  300$  and  experienced  schedule  slippage  factors  of  135$, 

Some  graphic  illustrations  are: 

Average  Unit  Cost 

Tactical  Fighters:  F-100  (2240  produced):  $  800,000 

F-105  (  835  produced);  $  4,000,000 

F-111A  (  160  produced):  $  8,500,000 

Strategic  Bombers:  B-4?  (2040  produced);  $  2,270,000 

B-52  (  745  produced);  $  7,200,000 

B-58  (  115  produced):  $12,400,000 

B-l  (in  development):  $30, 000, 000 ( estimated) 


Military  Transports:  C-47  (10500  produced):  4  100,000 

C-124  (  450  produced):  $  1,600,000 

C-130  (  338. produced):  $  2,866,000 

C-l4l  (  280  produced):  $  8,500,000 

C-5A  (  20  produced):  $30,000000 

j 

Table  1 

USAF  Aircraft  Cost  and  Quantity  Trends 

This  table  dramatically  demonstrates  two  well-known  trends:  a  rise  in 
unit  cost  and  a  decrease  in  the  quantity  of  aircraft  produced.  An  unfor¬ 
tunate  result  of  cost  growth  has  been  the  necessary  reduction  .in  the 
number  of  weapons  available  for  the  defense''  of  the  nation. 

Numerous  studies  of  the  causes  of  cost  growth  have  recently  been  accom- 

* 

plished  by  competent  teams  in  government  and  industry. _  From  a  risk 
viewpoint,  ah  amazing  pattern  of.  risk  initiation,  risk  growth,  and  risk 
transfer  emerges^-.especially  during  the  early  phases  of  the  system,  life 
cycle.  In  general,  programs  are  doomed  to  inevitable  cost  growth  and 
schedule  slippage  prior  to  the  establishment:  of  a  Program  Manager  (PM). 
There  is  ample  evidence  of  inattention  to  risk  In  all  phases  of  the 
acquisition  cycle,.  There  is  a  need  for  serious  attention  to  the  risks 
in  programs  in  each  of  the  following  areas : 

1.  Concept  Formulation.  For  more  than  two  decades  concept 
formulation  studies  have  pressed  for  the  last  10^  in  the  state-of-the-art 

.  and  the  earliest  possible  operational  availability  for  new  weapon  systems. 
Thus,  from  the  outset,  there  has  been  high  risk  of  exceeding  cost  and 
schedule  thresholds. 

2.  Threat  Evaluation.  Those  responsible  for  defining  operational 
requirements  are  motivated  to  insure  success  in  their  functional  area  by 

*See  items  5  through  15  of  the  Chapter  I  Bibliography. 


projecting  maximum  enemy  capabilities  rather  than  by  using  probabilistic 
estimates.  This  results  in  unrealistically  high  performance  goals  which 
increase  "risk  and  leave  .the  PM  little  or  no  allowance  for  meaningful 
tradeoffs . 

3'.  Initial  Program  Estimates ♦  Probably  the  most  significant 
single  factor  in  apparent  cost  growth  is  the  unrealistically  low  initial 
program  cost  estimate  submitted  to  Congress .  By  this,  approach,  which 
minimizes  the  chance  of  Congressional  disapproval,  planners  transfer  risk 
to  the  PM,  and  the  program  is-  in  serious  funding  trouble  from; the  start. 

4.  Source  Selection.  During  source  selection,  contractors 
minimize  their  chance  of  losing,  the  competition  by  submitting  the  most 
optimistic  proposals  regarding  the  unrealistically  high-  performance  and 
schedule  goals.  At  the  same  time,  they  tailor  their-  bid.  costs  to  the 
unrealistically  low  funding ^authorized  by  Congress .  Unavoidably,  the 
accumulation  and  growth  of  risk  at  this  point  is.  beyond  the  means  of  most 
Program. Managers  to  correct. 

5.  High  Technology  Programs.  Oyer  the  past  25  years,  pre¬ 
occupation  with  advanced  technology  has  produced  a  generation  of 
scientist-managers  who  have  heen  trained,  rewarded,  and- promoted  on  the 
basis  of  their  ability  to  manage  technical  innovation.  As  these  skilled 
professionals  came  into  dominance,  the  practitioners  of  conservative, 
cost-conscious  design  declined  in  numbers  and  influence.  Thus,  the  pro¬ 
grams  of  the  1970's  are  subject  to  substantial  risks  because  of  a 
propensity  for  overdesign  and  excessive  technology. 

6.  Engineering  Management  Systems.  The  pursuit  of  advanced 
technology  and  the  advent  of  the  scientist-manager  has  resulted  in 
laissez-faire  management  discipline  similar  to  that  which  promotes 


v>' 
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technological  breakthroughs  in  research  laboratories-  To  insure  minimum 
standards  of  management  discipline,  however,  a  vast  array  of  government 
imposed  management,  control,  and  information  systems  has  been;  imposed. 
These  systems  are  esqpensive  and  time  consuming- -said  may  be  counte'r- 
productive.  They  are  an  important  source  of  risk  and  inefficiency  in 
program  management. 

7.  Anticipated  Unknowns.  The -competitive  urge  to  be  the  leader 
in  producing -and  fielding  advanced  weapons  often  encourages  unwarranted 
optimism  regarding  programmed  milestones .  Too  often,  preoccupation  with 
schedules  induces- only  partial  completion  of  development  tasks  in  order 
to  reduce  the  chance  of  delaying  commitments  tbfull  production.  In 
this  way  substantial  risks  are  developed  and  transferred  to  later  phases 
of  the  acquisition  cycle. 

8.  Unanticipated  Unknowns.  During  the  I960' s  the  defense, 
acquisition  process. gravitated  toward  complete  system  specification 
based  on. paperwork  planning  and  paperwork  source  selection.  Military 
and  contractor  PM’ s  have  been  deluded  into  thinking  program  problems 
have  been  identified.  The  result  has  been  a  further  proliferation  of 
hidden  problems  which  grow  and  later  in  the  program  cycle  result  in 
breaches  of  cost,  schedule,  and  performance  goals. 

The  Semantics  of  Risk  Analysis 

Risk  analysis  is  not  an  established  science.  Many  of  its  terms  are 
borrowed  from  other  disciplines;  so  seme  of  the  key  words  will  mean 
different  things  to  different  people.  Accordingly,  it  is- necessary  to 
define  the  major  terms. 

"Risk"  itself  has  decidely  different  non-technical  and  technical 
meanings.  In  the  former  usage  for  example,  it  means  taking  a  chance  or 


RISK:  The  probability  that  a  planned  event  will  not  be 
attained  within  constraints  (cost,  schedule, 
performance)  by  following  a  specified  course  of 
action.  ' 


UNCERTAINTY:  Incoiiplete  knowledge. 


Risk  assessment  involves  the  process  of  risk  estimation 


-RISK  ASSESSMENT :  A  conprehensive  and  structured  process 

for-  estimating  the  risk  associated^  with  * 

5  a  particular  alternative  course  of 

action;  also  the  product  of  such  a 

■  ~  ^ 

process . 

Risk  management  is  concerned  with  the  generation  of  alternative 
courses  of  action  to  be  assessed  for  risk.  The  purpose  of  the  assess¬ 
ments  is  to  provide  the  manager  with  a  comparison  of  the  risks  involved 


RISK  ;MANAGEMENT :  The  generation- of  alternative  courses  of 
action  for  reducing  risk. 


Risk  analysis  combines  the  functions  of  risk  assessment  and  risk 

management. 


RISK  ANALYSIS :  The  process  of  combining  the  risk  assessment 
with  risk  management  in  an  iterative  cycle; 
also  the  product  of  such  a  process. 
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A  risk  analysis  is  a  joint,  effort  of  an  analyst-aanager  teas.  The 
manager  is  changed  with  the  generation  of  alternative  courses  of  action 
designed  to  develop  the  weapon  system  under  consideration.  The  function 
of  the  analyst  is  to  assess,  through  mathematical  jaeans,  the  risk  involved 
in  achieving  the  desired  product  as  a  function  of  the  cost  and  time— and 
to  indicate  the  subsystem  developments  which  most  affect  the  program  risk. 
The  manager  is  then  motivated  to  modify  the  courses  of  action  to  reduce  or 
avoid  the  risky  developments .  The  risks  associated  with  these  courses  of 
action  are  then  assessed  by  the  analyst.  This  assessment-modification 
iteration  proceeds  until  the  manager  is  satisfied  that  he  need  not  optimize 
the  alternatives  further- -at  which  time  he  selects  a  course  of  action  to 
use  in  proceeding  from  where  he  is  to  program  fruition.  The  procedure 
does  not  guarantee  an  optimum  course  of  action,  but  serves  to  apprise 
the  manager  of  the  risk  inherent  in  his  selected  strategy. 

In  order  to  accomplish  a  valid  risk  analysis,  those  who  are  accomp¬ 
lishing  the  assessment  need  access  to  planned  development  strategies. 

In  the  early  part  of  a  program,  analysis  is  best  done  by  either  the 
Service  staff  or  a  contractor  retained  to  study  the  development.  When 
the  Program  Office  has  been  established,  the  analysis  is  best  performed 
by  either  the  contractor  or  a  staff  assigned  to  the  Program  Manager.  In 
those  instances  where  the  risk  analysis  or  assessment  is  used  as  part  of 
a  presentation  to  higher  level  authority  tor  a  major  tradeoff  decision 
or  milestone  review,  an  independent  objective  evaluation  of  the  analysis 
should  be  performed. 


Scope  of  the  Study 


The  major  purposes  of  this  study  are  to  describe  what  should 

constitute  a  risk  analysis,  to  indicate  where,  when,  and  how  such  a 

risk  analysis  should  be  used,  and  to  establish  why  risk  analysis  as 

presented  in  this  study  can  be  an  effective  tool  in  reducing  the 

« 

excessive  cost  growth,  schedule  slippage,  and  performance  problems 
which  increasingly  plague  the  weapon  system  acquisition  .process. 
Additionally,  areas  in  which  follow-up  effort  and  research  are  required 
are  identified. 

The  scope  of  the  study  is  restricted  in  several  ways.  First,  con¬ 
siderations  of  national  priorities,  military  utility,  and  the  political 
environment  are  excluded  from  risk  analysis  as  it  is  recommended  herein. 
Secondly,  time  constraints  precluded  detailed  study  of  the  role  of 
contracting  in. program  management  and  its  effect  on  risk.  We  recognize, 
however,  that -meaningful  risk  management  is  possible  only  if  the  assessed 
risk  is  contractually  recognized.  This  means  that  follow-on  studies 
should  include  careful  attention  to  contractually  recognizing  assessed 
risk  in  ways  which  will  provide  the  flexibility  in  the  contract  instru¬ 
ment  needed  to  motivate  the  contractor  and  permit  desired  tradeoffs . 

Finally,  this  study  is  not  intended  to  be  a  technical  manual  or 
procedual  checklist  for  those  who  may  have  to  actually  perform  a  risk 
analysis.  Rather,  it  is  intended  to  provide  the  basis  for  management 
guidance  and  policy  in  implementing  risk  analysis. 

.y* 
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CHAPTER  II:  RISK  &  UNCERTAINTY 
■general  Considerations 

"v  - 

Risk  has  previously.  been  defined  to  be  the  probability  that  an 
event  will  not  be  completed  within  specified  time,  cost  and  per- 
formance^constraints  by  following  a  specified  course  of  action.  \ 

Uncertainty  has  been  characteri7ed  as  a  state  of  incomplete  knowledge. 
Thus  risk  and  uncertainty  are  not  synonymous..  Their  relationship  is 
probably  best  illustrated  through  the  example  shown  in  Figure  1, 


Risk 

I.Q  _ 

0  _ 


(a)  No  Uncertainty 


Cost 


Risk 


Risk 


Figure  1. 

Relationship  Between  Risk  and  Uncertainty 


\ 
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Consider  first  the  development  of  a  product  which  is  so  far  within  the 

V 

state-of-the-art  that  the  cost  of  development  may  ,be  ascertained  with 
certainty.  (Figure  la..)  Then  the  risk  associated,  with  its  development  is, 
either  zero  (if  the  cost  constraint  is  greater  than  the  actual  cost^Kor 
one  (if  the  cost  constraint  is  less  than  the  actual  cost)..  Thus,  where 
there  is  no  uncertainty,  it  is  known  in  advance  whether  or  not  a  project-, 
can  be  completed  within  a  specified  cost  constraint. 

Now  /Consider  the  development  of  a  product  in  which  there  are  uncertain¬ 
ties  involved.  Again,  if  the  cost  constraint  is  too  stringent,  the  risk 
is  one  (the  project  is  a  certain  failure);  if  it  is,  sufficiently  relaxed, 
the  risk  is  zero  (the  project  is  a  certain  success).  There  is,  however,  a 
middle  ground  in  which,  for  certain  cost  constraints,  the  risk  is  somewhere 
between  zero  and  one;  that  is,  there  is  some  chance  the  project  will  be  a 
success  and  some  chance  it  will  not.  The,  more  uncertainty  there  is  in  a 
project,  the  greater  will  he  this  middle eground  in  which  only  a  probabilistic 
estimate  of  success  may  be  made.  (Figures  lb  and  Ic.')' 

As  time  goes  on  and  as  money  is  spent  in  a  particular  project  more 
knowledge  may  be  obtained  and  therefore  the  uncertainties  may  be  reduced 
and  the  estimates  of  risk  may  improve.  However,  the  risk  of  completing 
the  project  within  the  original  cost  constraint  may  either  go  down,  remain 
the  same,  or  go  up.  In  other  words,  the  additional  knowledge  obtained  may 
show  that  the  project  was  less  expensive  them  originally  thought,  about 
the  same,  or  more  expensive.  This  same  discussion  could  apply  to  time  as 
well. 

The  initial  thrust  of  this  study  was  directed  toward  the  analysis  of 
"technical"  risk  and  the  technological  uncertainty  associated  with  it.  It 
quickly  became  apparent,  however,  that  "technical"  uncertainty  was  only 
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the  visible  tip  of  the  iceberg.  Submerged  beneath  the  surface  and 
frequently  having  a  far  greater  impact  on  program  objectives  and  success 
in  meeting  them  were  a  number  of  additional  uncertainties,  'none  of  them 
^purely  "technical"  in  origin  and  not  all  of  them  internal  to, the  program. 

We  have  classified  these  uncertainties  into  four  categories:  TARGET, 
TECHNICAL,  INTERNAL  PROGRAM  and  PROCESS.  There  is  some  overlap  among  these 
categories*  and  the  boundaries  between  them  must,  in  some  esses,  b-  arbi¬ 
trarily  defined.  It  might  be  more  accurate  to  describe  them  as  components 
in  a, ‘differentiated  continuum  of  uncertainty  rather  than  as  discrete 
categories.  Nevertheless,  the  essential  differences  between  them  are  clear. 
It  is  the.  experience  of  the  study  team  that  viewing  uncertainty  in  this  way 
sharpens  analysis  of  its  impact  on  the  weapons  system  acquisition  process, 
leading  to  an  enhanced  understanding  of  the  mechanism  connecting  uncertainty 
with  risk. 


Target. Uncertainties 

/Description . — The  term  "tarrot”  is  us.d  ~r.  this  repc.  t  to  refer  to  the  desire 

physical  and  performance  characteristics  which  a  weapons  system  must  have  to 
satisfy  a  given  need  or  requirement.  Target  also  refers  to  the"  desired  .cost 
arid  .-schedule  goals  established  for  the  development,  program.  Target  uncer¬ 
tainty  is  the  uncertainty  involved  in  reducing  a  need  to  cost,  schedule,  2nd 
performance  goals. 

Discussion. —Target  uncertainties  enter  the  weapons  system  acquisition  pro¬ 
cess  throughout  its  life  cycle  in  a  variety  of  ways.  These  can  be  generally 
summarized  as  follows: 

o  Uncertainty  concerning  the  nature  of  the  need  or  desired 


operational  capability. 


I 


o  Uncertainty  -introduced  through  Ithe  formal  .process:  of 
generating  requirements. 

o  Uncertainty  concerning  the  physical  and  performance 
characteristics  which  the  system'  must  -possess  if  it  is 
to  satisfy  the  stated  requirement, 
o  Uncertainty  of  cost  and  schedule  estimates. 

The  uncertainty  of  the  need  stems  largely,  from  the  vagaries  of  inter¬ 
national  relations,  the  intent,  and  capabilities  of  U.S.t  foreign  policy,  the 
nature  and  extent  of  enemy  threats  and  consequent  uncertainty  concerning 
the  nature  of  the  expected  operational  environment.  Even  when  the  need  is 
well  defined  and  clearly  understood,  uncertainty  can  be  introduced  through 
the  mechanics  of  converting  the  need  into  a  stated  requirement  for  hardware. 
This  can  begin  with  a  poorly  structured,  undefined  concept  formulatlon__ 
phase  and  run  throughout  the  life  6f  the  program.  Once  the  need  for  a 
particular  kind  of  weapon  system  is  determined,  it  becomes  necessary  to 
configure  it  ih\accordance  with  clear cut  physical  specifications — established 
target  criteria  of  a  relatively  precise  technological  nature.  No  program- can 
proceed  to  development  and  production  without  a  detailed  description  of/  what 
is  to  he  built.  But,  in  the  words  of  the  Assistant  Secretary  of  the 
Navy: 


The  whole  point:  of  the  development  process  is  to  get  something 
that  we  haven't  got;,  something  that  we  have  never  seen,  and  some¬ 
thing  .'that  we  really  don't  know  can  be  produced. ...  We  simply 
cannot  unambiguously  describe  before  the  development  begins,  or 
at  any  point,  in  fact  until  we  have  a  final  object,  what  it  is  we 
are -actually  buying,  (l) 

There  is  an  unavoidable  element  of  uncertainty  built  into  the  process 
of  translating  a-  relatively,  abstract  and  imperfectly-  understood  "need"'  into 
concrete  specifications.  The  system's  success  in- meeting  these  admittedly 
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uncertain  specifications  and  criteria  is  our  only  measure  of  its  operational 
worth  short  of-  full  scale  deployment;  yet.  full  scale  operational  deployment 
invariably  points  out  that,  these  criteria  were:  neither  complete  nor  entirely 

accurate.  In  many  cases  /the  validity  of  the  target  criteria  is  never  sub- 

/ 

jected  to  the  white  heat  of  'full  operational  exposure  and  is  therefore 
subject  to,  constant  reinterpretation. 

The  uncertainty  ana  unreliability  of  cost  estimates  have  a  direct  bearing 
on  risk.  These  -estimate s - r- which  are  invariably  inaccurate  in  the  early  stages 
of  the  acquisition  process  and  even  more  pronounced  where  the  desired  tech¬ 
nological  advance  is  greatest — go  into  tne  LOD  planning  machinery  and  tend  to 
be  "cast  in  concrete."  Congress  and  the  aerospace  contractors  learn  of  them 
and  use  them  for  their  planning  purposes.  Despite  Changes  made;  to  the  pro¬ 
gram  elements  in  the  interest  of  improving  the  technology;  adjusting  to  the 
threat ,_etc the  .early  cost  estimates  themselves  are  seldom:  revised..  Faulty 
estimates  in  the  planning  stages  ultimately  either  require  relief  from  the 
specified  cost,  schedule  and  performance  constraints  or  they  result  in  cost 
growths. 

Technical  Uncertainty- 

Description.  — Target  and  technical  uncertainties  kre  very  closely  related. 

To  distinguish  between  them,  one  must  separate  the  uncertainty  of  the  criteria 
used  in  solving  a  technical  problem  from  the  technical,  solution  itself  . 
Technical  uncertainty  treats  the  question  of  whether  a  system  can  be  developed 

v 

at  all,  within  any  time  frame  and  for  any  cost,  or  the  degree  of  difficulty 
which  will  be  involved  in  building  it. 

Discussion. — The  entire  concept  of  state-of-the-art  is  a  highly  subjective 
one.  There  is  considerable  difference  of  opinion  among  technologists  as  to 


wha z  is  meant  by  the  statement  that  something  is  ‘^technically  feasible"  or 
"within  the  state-of-the-art.  *'  Our  main  purpose  in  this  section  is  not  to 
become  engaged  in  a  semantic  argument  over  the  meaning  of  these  terms.,  but 
to  point  but  some  practical  considerations  arising  from  the  concept  of 
"state-of-the-art is"  Before  an  analyst  can  predict  the  difference  between 
what  Is.  available  and  what  is  considered  advanced  technology,  he  must  decicte 
on.. some  definition  of  the  term.  In  those  production  processes  which  are 
routine  and  repetitive  in  nature  there  is  understood  to  be  a  body  of  general 
knowledge  immediately  available.  Access  to  this  knowledge  is  easily  achieved. 
Development,  cn  the  other  hand,  is  concerned  with  one-of-a-kind  projects  such 
as  complex  weapon  systems.  The  state-of-the-art  in  such  advanced  development 
and  manufacturing  processes  nay  be  widely  dispersed,  may  involve  proprietary 
information  and  will  usually  vary  greatly  between  contractors  and  government 
laboratories.  It  is  in  these  latter  kinds  of  undertakings  that  the  degree 
of  technological  readiness  requiring  special  expertise  is  uncertain. 

For  the  last  several  decades  the  developments  in  U.S.  weapons  systems 
haye  reflected  an  engineering  "doctrine  of  quality"  which  stressed  advanced 
technology  at  the  expense  of  quantity.  (2,  p,  1$  3,  p.  64)  The  business 
and  political  environments  encouraged  this  doctrine  and  magnified  the  problem 
of  making  necessary  tradeoffs .  This  has  led  to  a  situation  of  technological 
myopia  which  has  (downgraded  any  effort's  to  develop  a  management  perspective 
v.^med  at  {1}  correctly  assessing  what  is  required  and  can  be  achieved  within 
dollar  and  time  constraints,  (.2)  formulating  technical  predictions  in  a  way 
that  will  permit  verification  and  (3)  budgeting  resources  to  accomplish  the 
technological  objectives.  The  assessments -of  the  past  have  been  extremely 
error-ridden,  useless  and  difficult  to  refute.  They  were  useless  and 
irrefutable  because  in  stating  that  a  certain  performance  was  physically 


possible,  they  often  ignored  the  important  time  and  cost  constraints.  Many 
cost  overruns  begin  with  optimistic  est.imft.tes  of  the  state-of-the-art. 

When  does  the  solution  to  &  technical  problem,  lie  beyond  the  program 
constraints?  When  is  a  problem  beyond  the  present  capabilities  of  the 
scientist  and  engineer?  The  difference  between  these  two  questions  is  real' 
and  lies  to  a  large  extent  at  the  basis  of  technical  uncertainty.  In  those, 
situations- where  unanticipated  problems  arise — problems  which  obvicusly 
cannot  be  forecast  in  advance  with  a  high  level  of  confidence— the  technical 
difficulties  which,  they  pose  may  not  he  accurately  assessed.  If  these 
problems  occur  at  a  time  when  hardware  is  being  built  and  tested,  the  design 
engineer  may  not  wish  to  consider  the;- problem  as  being  insoluble  for  fear 
that  his  peers  and  supervisors  might  downgrade  his  engineering  -ingenuity. 

The  Prograia\  Managers  in' both  government  and5  industry  often  react  to  technical 
uncertainty  by  designing  around,  it,  by  maxing  tradeoffs  and  finally  through 
submission  of  Engineering  Change  Proposals.  This  may  help  to  explain  why 
many  of  the  engineers  and  managers  interviewed  in  the  preparation  of  this, 
report  stated  that  it  may  net  be  possible  to  tell  if  a  technical  problem 
is  genuinely  insoluble.  The  human  factor  which  motivates  man  to  attempt 
the  inpassible— and  in  some  instances  succeed— also  causes  him  to  conceal, 
put-off,  or  uselessly  persevere  over  a  genuine-  "insoluble."  Even  here,,  it 
is  really  a  question  cf  time  and.  resources .  Who  is  to  say  that  given  enough 
time  and  money  the  problems  could  still  not  be  solved? 

This  complex  .interaction-  of  engineering  ingenuity  and  the  "state-of-the- 
art"  forms;  the  basis  for  many  problems  in  the  weapons  acquisition  process . 

The  inability  to  ^accurately  assess  the-  technical  feasibility  of  large-scale 
technological  endeavors  within- stated  cost  and  time  constraints  results  in 
changes  to  an  already  "slippery"  technical  baseline.  Thus,  purely 
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"technical"  problems  contribute  to  the  overall  uncertainties  of  development. 


Internal  Program  Uncertainty 

Description. -“Internal  program  .uncertainty  involves  those  uncertainties  that 
originate  within,  the  program  as  a-  result  of  the  way  in  which  it  is  organized, 
planned  and  managed.  The  boundary  between  internal  program  and  progress  is 
largely  ah  organizational  one;  internal  program  uncertainties  are  those  that 
>are  under  the  jurisdiction  of  the  Program  Manager.  Internal  program  uncer¬ 
tainty  is  the  uncertainty  inherent  in  selecting  -a  particular  method  or 
managerial  strategy  for  dealing  with  a  given  problem,  not  the  inherent 
uncertainty  of  the  problem  itself i 

Discussion. --Internal  program  uncertainty  enters  the  weapons  system 
acquisition  process  in  a  variety  of  ways.  These  generally  occur  chronolog¬ 
ically  as  follows: 

o  Uncertainty  of  initial  estimates  in  all  other  areas — process, 
technical,  and  target--inasmuch  as  they  impact  internal  program 
styling,  planning  and  management. 

6  Uncertainty  in-  selecting  among  various  acquisition  strategies . 
o  Uncertainty  in  program  management, 
o  Uncertainty'  of  program  outcome . 

The  first  three  areas  of  uncertainty  listed  here  are  closely  inter- 
coupled,  not  only  with  each  other  but  with  process  uncertainty  as  well. 
Perhaps  -the  most  difficult  aspect  of  analyzing  process  aha- internal  pro¬ 
gram  uncertainty  is  in  deciding  where  to  assign  the  role  of  uncertainty 
in  estimates  of  the  available  .resources. 

One  of  the'  earliest  "hard"  program  decisions  is  the  selection  of  ah* 
acquisition  strategy.  Uncertainty  concerning  which  acquisition  strategy 
to  use--total  package  procurement,  full. prototyping,  competitive 
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development  of  selected  components*  competitive  paper  studies,  etc.*- is 
amplified  by  the  uncertainty  in  early  estimates  of  the  resources,  needed 
and  resources  available,  considerations  which  we  will  discuss  in  our  treat¬ 
ment,  of  process'  uncertainties* 

.Having  selected  a  particular  acquisition  strategy*,  progr&m  styling  must 
be  established.  The  object  of  selecting  suitable  program  styling  is  to 
strike  the  proper  balance  among  the  basic  program  elements  of  cost, 

■schedule  and  performance.  Trie- balance  struck  among  these  by  internal 
program  styling  should  reflect  an  accurate  assessment  of  process  arid 
technological  uncertainties.  It  should1  be  emphasized  that  this  is  not.  a 
static  balance,  but  a  dynamic  one  which  changes  constantly  throughout  the 
life  of  the  program. 

Today's  turbulent,  environment  requires  that  flexibility  be  built  Into 
every  weapon  system,  program-  Efforts  on  the  part  of  the  government  and 
industry  to  definitize  the  system  prem&ttireiy  or  a  failure  to  make 
adequate  contingency  provisions  virtually  guarantee  program. imbalances. 
Flexibility  in  planning  and  management  is  a;  necessary  prerequisite  to 
government  and  industry  effectiveness  - 

Finally  we  must  consider  the/ -uncertainties  in  desired  program  outcome. 
The,  Program  Manager  Is  faced  with  a  large  -number  of  cqnpetihgr-and  often 
equally  valid-requirements  which  he  must  reconcile  through1  the  -mechanism* 
of  program  management.  Congress  is  primarily-  concerned  with  the  program's 
immediate,  fiscal  impact.  Planning;  staffs  at  the  Service  level  will  be, 
concerned  with  the  "ilities"— reliability,  vulnerability,  maintainability, 
etc.— and  with  projected  impact  oh  force  structure.  The  using  organi¬ 
zation  will  be  intensely  interest'd  in  performance  and  operating  costs. 

The  .military  Program  Manager  and  his  industry  counterpart  must,  in  one 


way  or  another,  reduce  the  uncertainties  generated  "by  tension  amang  these 
competing  requirements  if  program  success  is  to  he  insured.  They  must 
also  have  the  authority- -or,  as  a  minimum,  the  influence— to  do  so. 

-Process  Uncertainty 

Description. — Process  uncertainty  is  fundamentally  different  in  nature  from 
target,  technical,  and  internal  program  uncertainty.  Process  uncertainty 
originates  outside  the  program  hut  directly  affects  the  program's 
"critical  mass"  of  support .  The  Program  Manager  can  do  little' -to  alter 
process  factors.  His  only  reliable  tactic  for  minimizing  the  impact  of 
process  uncertainty  is  to  style  his  program  in  conformity  with  the 
realities  of  the  external  process  environment,  and  to  be  sensitive  to 
changes  in  that  environment.  The  uncertainties  here  are  much  broader  in 
scope  than  in  any  of  the  other  categories  and  concern  Service  priorities, 
other  weapons  programs,  roles  and  missions  debates,  DOD  policy,  the 
President's  budget. '.and  congressional  political  considerations . 

Discussion. — Discussions  of  process  uncertainty  should  consider  the 
following  subjects; 

a)  Uncertainty  that  resources  which  are  required  will  be 
available  when  needed  to  support  the  program. 

b)  Uncertainty  surrounding  the  criteria  to  be  used  in  the 
initiation  and  approval  of  program  changes., 

Resource  availability  centers  mostly  around  political  considerations. 
How  much  money  will  Congress  be  willing  to  obligate  for  the  program  in 
question?  How  do  the  resource  requirements  mesh  with  established  fiscal 
and  budgetary  planning?  How  much  control  and  surveillance  over  the  pro¬ 
gram  will  Congress  exercise?  What  will  be  the  relative  priority  among 
the  Services  for  the  monies  allocated  by  Congress?  This  last  question 
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points  out  one  of  the  single  greatest  process  uncertainties  i\n.  a  program. 
Where  the  likelihood  of  &  shortage,  of  funds  is  high,  managers  must  be 
provided  some  basis,  with  Which  to  style  their  programs  so  as  to  reduce 
the  inpact  of  stretchouts^  etc.  and  to  develop  contingency  plans..  In 
today’s  environment  there  is  great,  uncertainty  about  "which"  programs 
have  "what"  priorities-. 

The  uncertainties  associated  with  the  criteria  to  be  employed  ir. 
program  approval.,  contractor  selection  ana  in  the  review  and  evaluation 
of  on-going  efforts  focus  on  the  functions  and  activities  of  the  Defense 
Systems  Acquisition  Review  Council  (DSARC }.  Associated  with  this  organ! v 
zatipn  and  these  processes  there  is  an  adversary  role  to  be  played  at 
different  levels  of  management.  The  DSARC  must  examine  in  detail  and 
be  critical  of  those  program’s  being  promoted  and  advocated  by  the 
Services.  The  DSARC  has  been  organized  to  complement  the  Development 
Concept  Paper  system  and  to  advise  top  BOD  management  of  the  status  and 
readiness  of  a  major  program  to  proceed  to  the  next  phase  in  its  life 
cycle.  -When  events  and  parameters  in  these  programs  exceed' previously 


agreed  threshold  limits^  the  program  rut*.  ":c  reviewed  tc  determine  n  it 
should  be  permitted  to  go  forward  as  presently  configured.  Working  in 
this  capacity,  the  C:  n.cil  must  consider  important  issues  vrb:.c.i  are 
"non  advocate"  in  nature.  The  absence  of  a  data  base,  comparable  in 
size  and  quality  to  that  of  the  program  advocates,  and  the  absence  of  a 
staff  of  professionals  to  analyze  the  data  and  those  courses  of  action 
resulting  from  .it,  seriously  impair  the  efir- ;  x  I'D.  rC  '  - 

non-advocate  ro7te.  Considerable  uncertainty  1:  i<:.erc:y  sttc-crcri  ic 
whether  or  not  the  thresholds  established  for  each  program  are  realistic 
standards  of  measurement  for  deciding  when  a  program  should  progress  to 
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its  next  major  milestone,  arid  whether  or  not  the  threshold  has,  in  fact, 
been  attained.  Uncertainty  surrounding; t%e  DSARC's  decision -process  is 
typical  of  process  uncertainty. 

The  boundary  between  .process  arid  internal  progr 6s  uncertainty  is 
somewhat  arbitrary  because  they  are  both  intimately  concerned  with 
resource  allocation.  An  example  of  this  can  be  seed'  in  the  projected, 
funding  requirements  for  a  typical  system. 


Figure  2. 

Funding  for  a  Typical  System 


Figure  2a  shows  the  q/iginal  projected  .fiscal  requirements  which 
must  be  met  if  program  schedule  and  performance  goals  are  to  ;be  -achieved. 
Firgure  2b  shows  the  possible  -downstream  impact  of  this  year's  cut  in 
funding.  Whether  the  funding  cut  would,  in;  fact,  occur  was  a  process 
uncertainty.  The  way  in  which  it  will  impact  downstream  is-  in  part,  a 
process  uncertainty  and  in  part  an  internal  program  uncertainty..  Note 
that  the  total  program  expenditure  is  expected  to  increase.  This  is 
partly  due  to  inflation,  a  process  factor.  The  total  program  cost  also 
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increases  because  of  loss  of  program  efficiency,  and  contract  overhead — 
program  factors. 

Summary. 

The  weapons  Acquisition  Process  is  a  tremendously  complex,  turbulent 
network  of  activities.  Although  there  is< no ^satisfactory  static  model  of 
this  process,  there  are,  a  great  many  organizational  ar.d  cc;  .  r--  -  _  is 

that  assist  in  its  description.  The  breakdown  of  the  system  life  cycle 
into  the  phases  of  Concept  Formulation,.  Validation,  Full-Scale  Development 
and  Production  is  one  of  these  techniques.  Since  the  overall  acquisition 
process  is  characterized  by  high  uncertainty,  any  conceptualization  that 
clarifies  the  uncertainty  present  in  it  should  be  welcomed.  It  is  toward 
this  goal  that  the  categories  of  TARGET,  TECHNICAL,  INTERNAL  PROGRAM  and 
PROCESS  were  developed.  The  boundaries  between  categories,  are  not  always 
.clear  and  distinct,,  but  the  power  of  the  concepts  merits  examination  in 
more  detail.  The  next  chapter  will  examine  how  to  resolve  the  uncertain¬ 
ties  or  to  manage  around  them. 


CHAPTER  HI:..  RISK  JtASAGEWEHS 

The  preceding  chapter  introduced  the  four  categories  o?  uncertsinty 
•which  this  study  group  believes  encompass  the  prime  uncertainty  present 
in  a  weapon  system  development  program.  The  statements  concerning  the 


source  of  said1  uncertainties  are  illuminating.,  and  indeed  useful,  in  that 
many  of  the  sources  may  be  abate/ L  by  remedial  actions  of  the  BOD --ana  by 


the  Service  Secretaries  and  their  Civil  Service  and  Military  Staff  sub¬ 


ordinates.  But  such  changes  do  not  occur  overnight,  and  may  never  occur 
at  all  due-  to  other  constraints  extant  in  the  government .  The  problem 


then  remains- -how  to  resolve  the  uncertainties  into  known  quantities  or 


to  manage  around  them  if  this  cannot  be  done .  It  is  this  problem  we 
now  address. 

Our  objective  in  this  chapter  Is  to  introduce  some  techniques  available 
to  the-  developer-  in  his  task  of  generating  alternative  courses  of  action 
;to  be  used-'  in  bringing  his  weapon  system  development  program  to  fruition. 

In  conducting  a  risk  analysis,  one  ia  required  to  generate  an  initial  set 
of  alternatives,  assess  the  risk  in  these,  modify  them  to  reduce  the  risk, 
reassess  them,  and  continue  this--  process  until  one  alternative  may  be 
chosen  as  the  optimum,  end.  Both  the  initial  generation  and  the  subsequent 
modifications  may  be  -accomplished  by  using  the  techniques  presented  here-- 
and  others  since  this-  is  by  no  means  an  exhaustive  list. 

The  generation  of  alternative  strategies  for  the  acquisition  of  a 


particular  weapon  system  is  a  highly  innovative  process,  and  the  individual 
generating  such  procedures  can  only  depend  upon  his  experience  (which  in 
some  cases  is  seriously  lacking),  case  histories  of  similar  programs,  and 
his  and  his  associates’  imaginations „  There  is  a  dearth  of  information 
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in  the'  management,  literature  in  the  field  of  strategy  generation.  The 
major  focus  is  in  the  field  of  analysis.. 


_  Assuming  known  goals  and  clear  planning  premises,  -the 
first  step  of  decision  making  is  the  development  of  alter¬ 
natives.  It.  is  rare  for  alternatives  to  he  lacking  for  any 
course^  of  action;  indeed,  perhaps  a:  sound  adage  for  the 
manager  is  that,  if  there  seems  to, he  only  one  way  of  doing 
a- thing-,  that  way  is  probably  wrong.  What  the  manager  has 
probably  not  done  is  force  himself,  to  consider  other  ways,, 
to  open  his  eyes  and:  develop- alternatives;  unless  she  does 
so,,  he ‘cannot  know  if  his  decision  is  the  best  possible. 

(8;  152)  ’ 

The  Comptroller  General' s-Of fine  has  recommended  in  a  report  to  the. 
Congress  (5);  that  a  "decision-guide"  identifying  the  various  alternative 
acquisition  strategies  should  be  created— one  which  identifies  the 
features,  characteristics,  and» shortcomings  of  each.  Such  a  guide,  to 


•possibly  include  contracting  strategies  and  procedures,  would  be  immeas¬ 
urably  useful  to  anyone,  initiating  $  modifying,,  or  managing  a  weapons 

system  acquisition— and  especially  useful  to  the  : novice  Program  Manager 

.  / 

(as  rr.d-y  of  the  military  Pi-'/ *  arc).  Our  efx'orts  here  constitute 
'an  attempt  to  start  such  a  guide  „ 

The  intent  is  for  the  developer  to  ascertain,  based  on  the  definitions 


introduced  in  the  previous  chapter,  which  uncertainties  are  paramount  in 
the  program  under  consideration— and  to  generate  alternative  strategies 
for  development  -using  r/ne  techniques  mc,30  effective  against  those  uncer¬ 
tainties.  The  developer  may  then  proceed  to  conduct  risk  analyses  (as 
specified  in  Chapter  V)  on  these,  alternative  strategies,  change  them  as 


indicated,  by  the  analyses,  and  select  the  optimal  one  based  on  his 


utility  considerations , 

"The  fact-  that  one  does  not  need  to  make  decisions  in  the  face  of 
major  uncertainty  but  can  instead  take  steps  to  reduce  uncertainty  is 
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an  essential  quality  of  the  development  process." 


Management  Under  Target.'Uhcertaint.v 

Target  uncertainty  .'can  he  a  significant  factor  in  a  development  pro¬ 
gram.  It  has  been  frequently  pointed  out  that  a  weapons  system  rarely* 
if  ever*  is  used  for  the  original  purpose  for  which  it  was  conceived. 

Very  frequently  the  planned  target  does  not  fully  characterize  the  real 
need  that  .exists.  As  a  development  program  proceeds,  important  design 
requirements  are  recognized  by  both  the  developer  and  the  user  which  were 
not  previously  considered  by  either. 

Techniques  which  are,  effective  for  reducing  risk  arising  from  other 
forms  of  uncertainty  are  not  necessarily  appropriate  strategies  for  use 
when  target  uncertainty  is  a  key  factor.  For  example,  a  parallel  develop¬ 
ment  technique  for  a  system  or  subsystem,  which  is  valid  under  technical 
uncertainty,  is  not  highly  effective  when  target  uncertainty  prevails 
(see  the  .section  of  this  chapter  /entitled  "Parallel  Development"  for  a 
discussilon  of  this  concept).  Unfortunately,  those  actions  which  test 
and  clarify  critical  assumptions  in  the  specifications  may  not  be  carried 
out  until  very  late  in  the  development  cycle.  Thus  parallel  development 
may  have  to  be  gsix ied  very  far  along  before  sufficient  data  have  been 
gathered  to  identify  an  appropriate  choice.  This  may  result  in  very 
expensive  development  costs. 

Under  performance  target  uncertainty,  outcomes  tend  to  be  much  more 
desirable  when  the  following  actions  are  taken; 

I.  During  development  of  the  weapon  systems ,  continuing  effort 

is  expended,  in  refining  the  threat  which  the  system  is  being  acquired 

to  counter.  This  in  turn  should  allow  a  finer  specification  of  the 

performance  parameters  required  and  consequent  resolution  of  the 
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target  uncertainty  (the  requirements  stiffs  of  the  Mil t&ry  services 
are  charged  with  keeping  the  developer,  -whether  he  he  military  Program 
Manager  or  his  predecessor,  appraised  of  the  trend  in- performance  require¬ 
ments)  .  "The  further  downstream  In  time,  a  system  development  program  gets., 
the  more  costly  and  calamitous  are:  the  effect  of  changing  the  required1 
performance, 

2.  A  single,  ^promising  approach  is:  carried:  forward-'  inrougji  the 

development  cycle  to  the  point  where  it  can  he  tested  in  operational  or 

operational- like  conditions  before  a  final  conssitmsht  ,is  made  for 
'* 

production.  Once  preliminary  studies,  have  heed  accomplished,  a  course 
of  action  should  he  selected  which  concentrates  at  much  on  clarifying, 
the  appropriate  target  requirements  as  it  does  on  identifying  ;and  solving 
the  technical  problems. 

With  these  concepts  regarding  the  management,  of .‘"target  uncertainty  in 
mind,,  the  following  actions  are  proposed  as  potentially  useful  for  mini¬ 
mizing  or  reducing  the  risk  associated  with  target  uncertainty. 

1.  The  criteria  for  development  should  be  stated  in-  such  &  way 
(as,  for  example,  in  terms  of  upper  and  lower  limits  on  performance 
requirements)  that  tradeoffs  can  be  continuously  made  between-  technical 
design  and  operational  requirements  throughout  the  system- development. 

2.  A  Steering  Group  should  be  established  combining  represent¬ 
atives  of  the  development  management  team  and  the  user  to  review  design 
and  operational  requirement  tradeoffs.  Such  a  group  should  he  con¬ 
stituted  at  a  sufficiently  high  managerial  level  as  to  minimize  the  time 
required  to  effect  desirable  changes  (see  section  sniffled  "Uncertainty 
of  the  ‘..cess  for  Change  Approval"  later  in  this  chapter. 

"See  the  sss-li-n  of  this  chapter,  "Operational  Prototyping." 
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3.  Since  considerable  time  has  usually  elapsed  between  tie 
approval  of  stated  requirements  and  award' of  the -development  contract, 
i't  wculd.be  appropriate  to  establish  a  30-60  day  "cooling-off-"  period 
between  source  selection  and  the  contract  award.  In'  this  non-competitive 
environment,  the  development  and  Operational  segments  of  the.  Service  and 
the  Program  Manager  should  review  the  stated  requirements  with  ther  con¬ 
tractor  and/update  them  if:  necessary.  At  the  time  the  development  contract 
is  \awarded,  the  winning  contractor  has  supposedly  established  his1  capability 
to  deliver  a  'System  to  meet  the  main  features  of  the  stated  operational 
requirements.  Small-changes  in  the  requirements,  should  therefore  not 
invalidate  the  selection  of  that  contractor  for  the  program,  A  proposal 
such  as  this  will  probably  not  decrease  the  optimism  acknowledged  by 
industry  as  being  a  major  part  of  the  replies  to  BFP*s,.  but  should 
certainly  tend  to  diminish. the  number  and5  magnitude  of  Engineering  Change 
Proposals  submitted  subsequent-  to  contract  awarvd. 

4.  Uncertainty  concerning  the  estimates  of  time  and  cost  required 
to  develop  a  weapon  system  h&ve  been  identified  as-  target  uncertainties. 

Most  people  involved  with  weapon  system  acquistion  realize  that  these 
estimates  are  just  that,  but  they  are  treated  by  some  others  deter¬ 
ministic  quantities.  Their  probabilistic  nature  should  be  acknowledged 
by  quoting  "risk  profiles"  (as  in,  Figure  1,  Chapter  II)  for  weapon 
system  acquisition  courses  of  action  --and  if  a  single  number  is  required 
for  budgeting  or  other  such  purposes,  the  risk  associated  with  that  cost 
or  time  should  be  included.  The  "region  of  uncertainty"  'in  the  risk 
profile  may  be  decreased  by  increasing  the  accuracy  of  cost  and’  schedule 

^Chapters  IV  and  V  discuss  in  much  greater  detail  the  means  to 
obtain  these  profiles— and  their  uses. 
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estimates  (for  subsystem  developments)  input  to  the  risk  assessment 
which  produces  the  profiles.  Toward  this  end,  estimating  techniques' 
must  be  improved  and  seminars  or  courses  created;  to  instruct,  those 
charged  with  the  estimating  activity. 

Management  Under, Technical  Uncertainty 

The  technical  uncertainty  prevalent  in  a  development  program  may  be 
assessed  as  high  or  lew  by  answering,  the  question:  Do,  I  know-  how  to  build 
it— that  is,  do  I  know  how.  to  construct  and  mate  the  parts  and  establish 
quantity  production  procedures  to.  turn  out  large  numbers  of  the  item,  if 
that  is  required?  If  the  Answer  is  yes  then  the  technical  uncertainty 
is  low,  and  conversely.  High  technical  uncertainty  is  best  resolved  by 
high  order  hardware  proofing  activities  like  model  testing  ana  the-  three 
types,  of  prototyping.  Varying  degrees  of  technical  uncertainty  require 
different  levels  of  proofing  or  uncertainty  resolution.  Glenhan  (?) 
has  provided  a  classification  of  proofing  techniques  -which  ranges  from 
paper  analyses,  in  the  case  of  low  technical  uncertainty,  to  prototyping, 
where  technical  uncertainty  is  predominant.  Glenhan's  levels  are:  (l), 
paper  analyses,  (2)  design  review  by  experts,  (3)  focused  applied 
research,  (4)  model  testing,  and  (5)  prototyping.  ,They  are  all  described 
here  for  organizational  simplicity. 

If  very  high  technical  uncertainty  exists,  parallel  development  is, 
indicated.  (That  concept  Is  described  immediately  following  the  dis¬ 
cussion  of  proofing  techniques.) 

t 

Froofing  Techniques 

1.  Paper  Analyses.  This  technique  is  characterized  by  extensive 
use  of  mathematical  models  of  the  physical  world  representing  the 


environment,  within  which  the  .system  must  operate-  and  a  mathematical 
abstraction  of  the  system  itself.  A  wide  variety  of  hypothetical, 
designs  can- be  investigated,  and  at  least  preliminary  judgments  can 
-be  made  ori  the  suitability  of  alternative  designs  and  development 
strategies,  Analyses  =and  paper  studies  sire  most  useful  in  programs 
that  -lead  to  highly  specialized  products— where  there  is  little  or  no 
interaction  between  component  parts  30  that  they  are  sufficiently 
independent  of  each  other  to  allow  disjoint  treatment .  One  of  the 
major  approximations  required  in  the  use  of  mathematical  models  involves 
simplification  of  the  interrelationships  between  components,  wherein  a 
great-  deal  of  realism  is  .sometimes  lost— especially  in  complicated 
programs . 

2.  Design  Review  by  Specialists,  If  a  development  strategy  or  set 
of  alternative  strategies  is  proposed,  a  design  review  team  of  experts  in 
varied  specialties  can  be  used  to  assess  the  degree  of  uncertainty  present- 
in  their  particular  areas  of  expertise.  This  is  again  a  device  which 
avoids  the  expense,  of  actually  building  prototypes  and,  is  a  concept  which 
has  been  extensively  employed  during  the  60's,  A  proposed  method  of 
attack,  when  critiqued  by  experts  in  fields  such  as  power-plants  > 
materials,  structures,  performance,  human  engineering,  and 
reliability  may  be  modified  to  conform  to  what  is  known  based  on. 

.the  experience  of  the  expert,  reviewers.  This  procedures  is  iterative, 
must  start  off  with  a  "straw  man,"  is  highly  susceptible  to  human 
frailties  such  as,  optimism  and  cynicism,  and  will  not.  necessarily  produce 
an  optimal  course  of  action.  The  parpchial  interests  of  each  "-ility" 
expert,  and  the  personality  differences  between  them  can  lead  to  unbal¬ 
anced  programs.  There  is,  in  addition,  the  uncertainty  as  to  whether 
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all  the  necessary  "-ility"  experts  have  been  included  in  the  review 

process,  There  ns  on  record  the  case  of  an  anti-mortar  radar  developed 
{ 

for  the  Arays  which  met  all  design  criteria  and  operated  perfectly — until 
deployed  operationally  in  Korea.  It  turned  out  that  the  radar’s  operating 
frequency  fexactly  duplicated  the  mating  call  of  a  moth  found  locally  in 
great  abundance and  the  antenna  was  completely  blocked  out  by  enormous 
swarms  of  romantically  inclined  insects.  No  entomologists  were  included 
on  the  design  review;  team. 

3*  Focused  Applied  Research*  ‘The  first  type  of  uncertainty 
resolution  involving  actual  hardware  construction  .'is  called  "focused 
applied  research."  Once  a  basic  design  has  been  selected  for  development1 
there  will  be  many  questions  surrounding  the  materials,  manufacturing 
techniques,  or  other  technical  inputs  which  may  be  resolved-  by  controlled 
laboratory  tests.  These  tests  share  with  analysis  the  quality  of  abstrac¬ 
tion.  The  test  of  the  strength  of  a  piece  of  material  is  usually  con¬ 
ducted  with  a  standard  test  sample  of  material.  The  shape  is 
determined  by  the  testing  .procedures  and  the  strength  will  be  measured 
according  to  some  standard  method.  However,  in  actual  usage  the  material 
will  be  shaped  differently,  subjected  to  different  stress  levels  and 
dynamics,  and  will  no  doubt  have  peculiarities  of  fabrication.  The 
ability  of  a  designer  to  translate  the  test  results  into  valid  information 
concerning  the  actual  design  will  determine  the  usefulness  of  the  tests. 
Such  focused  applied  research  results  should  be  more  cautiously  applied 
when  the  product  differs  greatly  from  previous  ones. 

4.  Model  Testing,  The  next  category  for  resolving  technical 
uncertainty  involves  model  testing.  It  differs  from  focused  applied 
research  in  zh&i  a  model  represents  a  partial  synthesis  of  components J 
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and  it  differs  from  prototype  testing  (discussion  follows)  in  that  it 
seriously  abstracts  from  the  final  product. 

rThe  use  of  a  model  in  development  follows  from  a  recognition 

that  the  complexities  of  a  design  prohibit  complete  analysis,  perhaps 

h 

because  of  the  complex  interaction  of -components  or  because  the  second 
order  effects  are  inport ant .  These  models  must  be  tested1  in  a-  simulated 
environment  and  because  of  this,,  a, calibration  of  the  environment  is 
required-.  To  the  degree  that  the  environment  is  known  .arid., understood,  the 
calibration  is  relatively  straightforward.  We  are  reasonably,  confident, 
for  instance,  of  test  work  in  subsonic  wind  tunnels  because  of  the  wide 
range  of  past  experience  with  the  translation  of  wind  tunnel  results  to 
actual  practices  These  tunnels  arid  experiments  are  generally  "well 
calibrated."  When  we  seek  to  extend  our,  results  beyond  our  experience, 
however,  the  problems,  of  calibration  increase  and  our  confidence  in  the 
tunnel  results  should  decrease.  Serious  problems  can  be  expected  in 
interpreting  the  results  when  significantly  different  (advanced)  products 
are  being  developed. 

5.  Prototyping*  The  most  expensive  but  most  realistic  category  of 
uncertainty  resolution  involves  the  use  of  prototypes  in  testing  programs. 
The  word  "prototype"  has  many  connotations,  particularly  in  connection 
with  military  developments,  but  for  present  purposes  .&  pr.cj^i^' will  be 
a  full  sized  or  nearly  full  sized  model  that  can  be  i-este<3^|ig&$he  true 
physical  environment  in  which  the  final  product  will  he  used.  Because 
it  represents  a  first  approximation  of  a  product  and  is  expected  to  be 
changed  as  a  result  of  testing,  it  will  generally  (except  possibly  in 
the  case  of  "production"  prototypes)  be  built  with  a  minimum  of 
specialized  capital  equipment  so  as  to  save  both  money  and  time. 
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Moreover,  a  prototype  can  be  a  subsystem,  a  collection*  of  subsystems,  or 
a ^complete  system  depending  upon  the  needs  of  the  development  project. 

Prototypes  do  overcome  some  of  the  disadvantages  of  lower-order 
proofing  techniques.  Since  prototypes  are  usually  full  size,  results 
obtained  in  tests  do  not  have -to  be  scaled.  And  since  they  operate  in 
the  environments  in  which  the  final  products  will  be  used,  there  are  no 
calibration  problems. 

The  DOD  Project  Hindsight  Study  indicated  that,  lit  many  weapon  system 
development  programs,  approximately  one-third  of  the  new  scientific  and 
technological  information  needed  to  satisfy  system/program  requirements 
was  generated  after  contract  initiation.  One:  of  the  most  powerful  tools' 
to  demonstrate  that  this  information  is  well  in  hand  is  that  of  proto¬ 
typing.  Normally  a  prototype  of  one'  sort  or  another  will  coincide  with 
a  development  milestone-express  or  implied. 

Peck  and  Scherer  (13)  indicated  that  use  of  prototype  testing  and' 
multiple  approaches  tended  to  decrease  the  time  required  to  gain  program 
fulfillment — for  a  specific  performance  vector.  These  were,  however,  pro¬ 
grams  which  included  a  high  degree  of  technological  uncertainty,  which 
probably  would  not  have’  been  resolved  satisfactorily  by  paper  studies 
or  sequential  attempts  to  develop  quantum  technology  jumps  which  are 
the  complements  of  prototyping  and' parallel  approaches.  The  time 
compression,  however,  is  not  free— in  that  program  funds  need  be 

i 

allocated  against  the  efforts.  They  proposed  a  functional  relationship 
as  follows  based  on  studies  of  major  weapons  systems  acquisition  program^ 
where  technical  uncertainty  was  considerable: 
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Functional  Reltionship  of  Prototyping  Funds  to 
Project  Development  Tine 
(For  a  particular  performance  vector) 


The  rationale  for  the  curve's  shape  being:  (l)  for  the  asymptotic 
nature  at  a  minimum  time-fthere  is,  in  any  program, » a  serial  sequence 
of  operations  which  must  be  performed  regardless  of  whether  one  proto¬ 
types  of  not,  and  (2)  for  the  intercept  on  the  abscissa — no  prototyping 
means  no  funds  devoted  to  that  activity,  but  less  extensive  hardware 
proofing  studies,  which'  are  the  alternative,  do  not  always  yield  valid 
results  due  to  the  approximations  and  errors  made  therein — resulting 

in  rework  and  restudy  to  discover  the  faulty  components  when  the  program 

» 

end  item  is  finally  delivered.  (3)  The  convexity  toward  the  origin 
was  an,  empirically  observed  phenomenon^ 

There  are  three  basic  prototyping  philosophies,  dependent  upon  the 
relative  time  that:  the  prototype  is  constructed  and  the-’  reason  for 
which  it  is  required.  They  8re*  development  prototyping  (brassboarding, 
breadboarding),  production  prototyping, ;and  operational  prototyping, 
a.  Development  Prototyping-  (brass-  or  breadboarding) 

The  objective  for  the  use  of  prototypes  in  development  is 


basically  to  provide  engineering  data  complementary  to  that  provided  by 
analysis,  design  review,  and  focused  applied  research.  Development 
prototypes  are  normally  constructed  and  tested  during  the  engineering 
development  phase  of  a  system  (e.g.:  the  AWACS  radar  set)  and\  can  vary 
from  a  breadboard  of  a  subsystem  to  a  complete  flying  prototype. 

The  usefulness  of  the  development  prototype  'Is 
greatest  if  it  can -be- built 'and  tested^  with -only  <au 
fraction  of  the  engineering  effort  (and  time)  required 
for  the  production  prototype .  The  objective  should  be 
to-  select  -the-  f undaaeritai-or.  :key  techMcalv(problemsi 
that  are  not  subject ‘With  assurance  jto  analysis  alone, 
and  to  design  and  build  equipment  that  will  provide 
the  necessary  engineering  data.  Formal/engineering 
of  the  type  necessary  for  a  production-release  can 
often  be  avoided,  .and'  features  should'/be  omitted  if 
they  are  not  essential,  to  meeting  the  purpose  of  the 
development  prototype .  The  determination  of  the 
objectives  and  resulting. characteristics  of  this  type 
of  prototype  requires  keen  engineering  judgment  if  it 
is  to  be  most,  useful.  Careful  review  is  necessary  to 
ensure  that  it  is  built  to  meet  a  genuine  need  that 
cannot  be  satisfied  -*ith  less  expensive  means,  aha  that 
the  prototype  does  not  "grow"  with1  the  addition  of 
costly  features  unnecessary  to  its  fundamental  purpose. 

ih)  .  . 

b,  Induction  Prototyping 

The  objective  for  the  use-  of  production  prototypes  prior  to 
production  commitments  is  to.  provide  information  on  producibility.  Here 
the  .prototypes  are  constructed  near  the  end  of  the  development  program, 
looking  as  close  as  possible  like  the  first  production  article .  Not 
only  is  the  prototype  itself  evaluated  but  so  are  the  tools  which  created 
it.  In  production  these  tools  are  the  production  processes-*  procedures^ 
and  organization  that  will  ultimately  produce  the  system.  Here  the- 
major  issues  are  producibility  and  production  costs,  { 

A  major  purpose  of  building  and  testing  a  production 
prototype  is  to  establish  with  assurance  that  the  engi¬ 
neering  design  is  qualified  for  a  production  release. 


The  engineering  drawings  and  specifications  eon- 
trolling  the  manufacture  should  in  general  be 
formal  and  thorough,  and  should  represent  as 
accurately  as  possible  the  concept  of  the  equip¬ 
ment  that  is  to  be  deployed.  The  technical  effort 
that  is  required  for  such  a  formal  release  is  sub-' 
stantial  and  involves  considerable  time  and  cost.  (4) 

c.  Operational  Prototyping 

The  objective  for  the  use  of  operational  prototypes  is,  to 
establish  the  feasibility  of  military  utilization  in  the  fields  of  per¬ 
formance^  maintainability,  safety.,,  etc.  Paper  studies  in  these  areas  are 
only  tentatiye,ar.d!  advisory,,  suggesting  whether  operational  prototyping 
is  necessary.  This  prototype  concept  is  effective  against  target, 
uncertainties  and  demonstrated  capabilities  may  be  assessed  as  suffi¬ 
cient  or  not  when  the  prototype  is  tested .  The  Havker-Siddely  P-il27 
(Harrier)  VTOL  closer  support  aircraft  now  being  tested  by  the  U.Si 
Marines  under  combat  conditions  constitutes  such  an  operational  prototype-. 
The  Marines  are  attempting'  to  establish  whether  or  not  a  sufficiently 
improved  close- support  capability  justifies' buying  the  aircraft  in 
numbers . 

Parallel  Development 

Where  a  quantum  jump  in  technology  is  required,  parallel  development 
by  different  organisations  of  duplicate  research  and  engineering 
toward  a  common  goal,  whether  it  be  paper  analyses  or  hardware,  is  a 
technique  to  be  considered.  The  rationale  is  that  two  or  more  groups 
trying- to  achieve  a  goal  in  different  ways  will  lead  to  a  more 
desirable  product  than  if  only  one  group  were  to  attack  the  problem 
in  its  own  way.  The  number  of  parallel  developers  can  be  decreased  as 

Abernathy  indicates  that  the  marginal  gain  in  learning  decreases 
with  an  increase  in  the  number  of  parallel  developers,  however. 
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estimates  improve  and  the  required  technology  ‘is  seen  to  be  more 
attainable.  In  the  aiomic-bpmb  project*  one  of  the  most  spectacularly 
successful  military  projects  the  United  States  BSis- -ever  undertaken,  the 
parallel-path  strategy  was  employed.  James  Sonant  \wrote  in  &  letter  to 
Bush  on  May  14,  1942: 

i 

All  five  methods  will  be  entering  very  expensive  .pilot 
plant  development  during  the  next  six  months,  ...(But)  while 
all  five  methods  now  appear  to  be  about  equally  promising, 
clearly  -the  time  to  production  ...  by  the  five  routes  will 
certainly  not  be  the  same  but  might  vary  by  six  ninths  or  a 
year  because  of  unforeseen  delays.  Therefore,  if  one  .discards 
one  or  two  or  three  of  these  methods  now,  one  may  be  betting 
on  the  slower  horse  unconsciously;  (12) 

If  the  early  .stage  development  costs  are  small  and  the  expected 
decrease  in  technical  uncertainty  large,  it  pays  to  run  parallel  projects. 
And\  if  the  competing  projects  Jure  similar  in  their  estimated  cost  and 
performance  because  there.'  is  little  data  on  which  to  base,  estimates,  but 
they  are  quite  different  in  design,  then  it  certainly  pays  to  run  several 
projects  in  parallel.  The  parallel-path  strategy  is  rational  when  time 
is  important.  When  time  is  not  pressing,  the  uncertainties  which  call 
for  the  parallel~path  strategy  if  development  is.  attempted  now  might 
better  be  interpreted  as  a  signal  that  development  should  not  be  ‘ 
attempted  until  more  background  research  has  been  done. 

An  additional  wrinkle  has  been  added  to  the  parallel  development 
concept  in  the  suggestion  to  pursue  "Parallel  Undocumented  Development" — 
which  was  very  favorably  received  by  the  GAO.  (5, 10 )  There  is  little  new 
in  the  concept  and  it  merely  realizes  that  the  less  paper  work  required 
from  a  contractor,  the  more  resources  he  is  able  to  devote  to  the 
development  problem  at  hand.  A  quote  in  Mr.  Hash’s  article  is  worthy 
of  note,  however: 
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. .. in  terms  of  -total  system  cost,  parallel  development 
maynot  be  -excessively  burdensciie.  George  Schairer  of  the 
Eoeing  Company  has  estimated  th«&  parallel  development 
through  the  prototype  phase  adds,  6  percent  to  the  total 
.cost  of  an  aircraft .program  with  a  contemplated  production 
of  500  planes.* 

Other  Considerations 

Technical  uncertainty  is  often  increased  by  the-  source  Selection 
process  through  apparent  technical  transfusion  prior  to  source  -selection. 
This  apparent  transfer  of  knowledge  gives  ri'.se  to  the  conclusion  that  any 
of  the  several  contractors  can  build  the  proposed  vweapon  at  the  current 
state.-of-the-art..  'However,  we  "conclude  that  whatever  actual  technical 
transfusion  exists  at  this,  point  tends  to  be  superficial  and  is.  not 
sufficient  to  makt;  -all  proposals  really  -technically  equal  because  the 
technical  capability  of  these  contractors  is  not  the-  same.  The  result  is 
to  transfer  the  ei^hasis  in  source  selection  away  from  technical  uncertainty 
issues  onto  lowest  cost  bids.  We  suggest  that,  the  DOD  reinforce  current 
policies  to  prohibit  the  government  from  assisting  in  technical  trans¬ 
fusion,  before  the  source  selection, :deeision.  This,  may,  for  example, 
preclude  C9%etitive  negotiation.. 


Technical  .uncertainty  in  some  programs  could' be  reduced  if  all  possible 
subcontractors'  were  made  available  to  the  winning  prime  contractor.  .Sub¬ 
contractors  are  normally  "locked  up"  by  the  prime  contractors  during 
competition.  This  precludes  some  prime  contractors  from  being  capable  of 
state-of-the-art  performance  in  some .areas.  We  should  institute  procedures 
whereby  the  government  has  complete  choice  of  subcontractors  to  work  with 

*The  Role  of  Competition  in  Aeronautics,  The  Wilbur  and  Orville 
Wright  Memorial  Lecture  of  the  Royal  Aeronautical  Society,  Dec  5,  1968 5 
quoted  by\R.,  C.  Nash  (10) 
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the  winning  prime  contractor.  Such  choice  should  include  the  option 
to  reject  a  subcontractor  who  may  have  "te&med-up"  with  the  selected 
prime  during  competition,  as  well  as  to  select  the  best  subcontractor  to 
replace  the  one  dismissed.  One  method -of  doing  this,  of  course,  involves 
identifying  those  subs  whose  products  are  desired  by  associate  contracts 
and  providing  the  subcontractors’  products  to  the  prime  as-GFE. 

Unanticipated'  Technical  Problems 

One  of  the  biggest  -problems  inherent  in1  programs  with -a  high  degree  of 
technical  uncertainty  is  the  inability  of  the  developer  to  specify  all  the 

4 

uncertain  component  development  problems.  They  are  frequently  encountered 
during  development,  or  after  production  (e.g. ,  TEX -swing  wing  pivot),  and 
have  bqen  termed"  "unknown  unknowns"  by  the  AIA  in  its  studies  since  they 
cannot  be  foreseen  when  development  is  started.  As  they  appear  during 
the  development  cycle.,  they  frequently  cause  performance  degradation, 
schedule  slippages,  and  cost  overruns.  The  only  way  they  can  be  handled 
is  to/ identify  them  as  they  occur,  and  then  to  resolve  them  in  accordance 
with  such  techniques  as  hardware  proofing  and  parallel  development.  A 
risk  analysis  as  proposed  in  this  report  cannot  identify  "unk  unks , " 
Funding  of  seme  sort  is  required  to  pay  for  the  resolution  of  such 
technical  "unk  unks"  once  they  are  surfaced,  but  timely  discovery  is  of 
paramount  importance,  A  good  cost,  schedule,  and  performance  parameter 
tracking  system  will  indicate  the  symptoms  of  surfacing  "unk  ’inks,"  and 
it  remains  for  the  program  developer  to  identify  the  precise  cause. 

A  reporting  system  which  indicates  the  current  estimates  of  final 
cost,  schedule  and  performance  and  the  current  status  of  the  same  items 
is  necessary  to  inform  the  PM  of  the  progress  and  status  of  his  systems, 
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and  to  aid  in  the  identification  of  “unk  unks."  If  progress  is  not 
preeeeding  as  scheduled  the  indication  is  that  some  previoiisly  unforeseen 
problem  area  is  rearing  its  head,  and’ identification- procedures  should  be 
initiated.  Spending,  or  even  budgeting}  in  excess  of  that  scheduled; 
excessive  time  spent  on  activities  in  comparison  to  that  scheduled;  and 
comparatively  slow  progress  in  approaching  required  performance  parameters 
are  ^danger  signals  Which  will  be  sounded  by  tracking  the  current  system 
status . 

Regardless  of  the  parameters  reported}  the  system  used  to  do  the 
reporting  to  the  PM  should  be  the  same  as  that  used  within  the  contractor’s 
own  management  information  system.  Rather  complete  latitude  is  provided 
for  information  systems  to,  be  used  for  project  control  by  the  current  DODD 
5C00.1  (Lj  Jul  "  DOpI  7000.2,  and  other  applicable  directives;  and  as 
long  as  the  FM  is  getting  the  information  he  requires,  there  is  no  need  for 
duplicate  information  systems,  reformatted  reports,  or  the-  like.  The  con¬ 
tractor’s  own  system  which  he  has  designed  and  understands,  and  mere 
important — uses  himself,  Is  the  best  one. 

Many  displays,  have  been  generated  to  allow  management  to  monitor  the 
progress  of  programs  Under  their  control,  but  most  suffer  from  the  same 
deficiency— they  are  confined  to  two  dimensions.  They  can  display  work 
completed  against  time,  or  funds  spent  agairist  time;  but  little  corre¬ 
lation  between  these  two  highly  dependent  variables  has-been  shown.  One 
of  the  major  weapons  systems  now  under  development  uses  the  cost, 
schedule,  and  performance  data  provided  by  the  contractor  in  response 
to  DODI  7000.2  to  display  cost  and  schedule  as  a  function  of  time  in 
the  following  manner: 
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Cost/Schedule/Performance  Parameter  Tracking  Graph 

a.  Abscissa  -  time  initiation  of-  development'  of  the  subsystem 
being  tracked.  Here  a  subsystem  is  used  in  the  sense  of  a 
specific  set  cf  Worfc  packages  as  specified  by  the  work 
breakdown  structure. 

b.  Ordinate  -  cumulative  dollars. 

i- 

c.  Curve  A  -  cumulative  budgeted-  cost  of  scheduled  work  packages 
included  in  this  tier  level  item.  This  is>  of  necessity  a 
monotoni'cally  non-decreasing  curve  and  terminates  at  the  end 
item  availability  date. 

d.  Curve  B  -  the  actual  cost  of  work  performed  on  this  subsystem 
which  represents  money  either  paid  or  committed  to  the  con¬ 
tractor  due  to  completion  of  work  packages  (the  work  packages 
are  assumed  to  be  satisfactorily  completed}  that  is,  the 
performance  specifications  are  met.) 


e.  Curve  C  ~  the  "earned  value11  in  this  subsystem,  item  -  the 
'budget"  value  of  completed  work  packages  in  the  subsystem.. 
The  B  and  C  curves  consist  of  connected  points  reported  on  a  frequent 
(monthly)  basis.  The  relative  vertical  orientation  of  the  latest  points 
leads,  to  the  analyses  presented  in  the  rows  of  matrix  below; 

t 

A:  Budget  Cost  for  Work  Scheduled  (BCW3) 

B:  Actual  Cost  of  Work  Performed  (ACW’P) 

C,;  Budgeted  Cost  of  Work  Performed  (BGWP);  "earned  value" 
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Matrix  Analysis  of  the  Cost/Schedule/Performance 
Tracking  Graph 


The  situation  depicted  on  the  graph  actually  existed  in  a  current  weapon 
subsystem  development.  With  just  curves  A  and  3  represented,  it  would 
appear  that  funds,  were  not  being  spent  as  fast  as  budgeted — but  the 
addition  of  Curve  C  indicates  that  the  planned  number  of  work  packages 


have  also  not-  been  completed.  The  actual  situation  is  not  optimistic 
but  is  definitely  a  cost  overrun  and  possibly  a  schedule  slippage  in 


addition.  The  possibly'  is  due  to  the  fact  that  work  packages  might  have 
actually  been  conpleted-  which  were  not,  a  priori,  scheduled  to  he  done  at 
this  tine — and  others,  which  were  budgeted  to  be  done,  could  have  been 
deferred.  If  these  latter  are  completed  later  on  at  a  lower  cest  than 
anticipated,  the  schedule  slippage  might  not  be  real. 

The  approximate  amount  of  time'  involved  in  schedule/  slippage  (or 
advance may  be  obtained  by  converting  the  dollar  difference-  between 
curves  A  and  C  to  time  by  dividing  the  dollars  by  the  current  rate  of  con¬ 
tractor  earning  (dollars  per  time). 

This  is  but  one  example  of  a  cost/ schedule/perfc-rmance  tracking  (con¬ 
trolling)  system  that  allows  revelation  of  anomalies  appearing  in  the 
program. 

Management  Under  Internal  Program  Uncertainty 

Obviously,  -flexibility  is  the  key  in  combating  the  probleas  arising 
from  internal  program  uncertainty.  If  the  developer  is:  unsure  that  the 
development  course  of  action  that,  he  has  selected  is  indeed  tha  appropriate 
on$  to  counter  the  problems'  in  his  program,  he  must  maintain  the  capability 
to  /change  the  enphasis  of  his  strategy/.  If  his  !,unk  unk"  discovery 
process  reveals  a  large  technical  uncertainty  rearing  its  head,  he  dust 
be  prepared  to  shift  strategies  and  adopt  some  high  order  of  hardware 
proofing,  maybe  even  parallel  development.  Such  a. -course  wa3  necessary 
in  the  case  of  the  60-foot  armored  launched  bridge  developed  by. the  Army. 
They  already  had  a  40-foot  version  and  proposed  to  build  upon  it  by 
changing  the ,material  used  in  constructing  the  bridge  carried  by  the 
tracked  vehicle;  frpra  steel,  to  aluminum-.?  the  centers  of  gravity  a.nd  total 
weights  were  to> remain  essentially  the  same.  This  was  thought  to  be  a 


rather  lew,  technology  uncertainty  program  until  someone  questioned  the 
ability  of  the  contractor  to  weld  aluminum  in  the  sizes  and  shapes 
specified-rib  had  never  been  done  except  on  small  parts  under  laboratory 
conditions.  Immediately  the  program  assumed  high  technical  uncertainty 
and  a  development  prototype  consisting*  of  welded  aluminum  members  was 
scheduled.  Additionally,  a  fallback  position  calling  for  riveted  con¬ 
nections  was  planned.  As  it.  turned  out,  ther'  aluminum,  welding  on  the 
prototype  went  well,  the  uncertainty  in  the  program  was  reduced  through 
the  knowledge  gained,  and  the  fallback  position  was  unnecessary.  Both 
the  institution  of  a  component  prototyping  strategy  during  developments 
and  the  scheduling  of  the  fallback  position  were  examples  of  the  flexi¬ 
bility  required  in  the  program. 

To  counter  the  uncertainty  in  program  planning,  separate  (parallel) 
analyses  or  design  reviews  by  specialists  are  possible  useful  techniques 
under  program  uncertainty— especially,  when  conducted  early  in  the  devel¬ 
opment  process.  The  insights  gained  by  examining  different  proposed 
coureses  of  action  for  system  development  will  indicate  whether  there 
is  a  great  deal  of  program  uncertainty  due  to  program  styling  and 
planning;  If  more  than  one  expert  organization  indicates  a  similar 
strategy,  then  program- uncertainty  due  to  the  uncertainty  of  the 
approach  method  is  reduced.  The  converse  is  true  if  widely  variant 
approaches-  are  suggested.  When  there  is  uncertainty  concerning  the 
methods,  management,  or  costs  associated  with  actually 'building  the 
production  article,  production  prototyping  might  be  indicated.  This 
process  is  extremeliy  expensive  however,  and  should  be  clearly  justi¬ 
fied  in  each  case. 
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The  central  figure  in  the.  weapon  system,  acquisition  process  is  the: 
Program  Manager.  It  has  been  suggested  that  most  of  the  ills  of  the 
acquisition  process  could  he  cured  by  selecting  better  Program  Managers. 
While  this  recommendation  would  help,  the'  situation,  we  find  tbs  PM's 
responsibility  greatly  exceeds  his  authority.  He  is  responsible  for  his 
program,  but  he  is  quite  often  the  victim  of  an  ill-defined  target, 
inadequate  resources,  and'  a  changing  process.  In  short,;,  many  of  his 
problems  are  caused  by  factors1  well  beyond  his  control.  One  way  to  reduce 
these  problems  is  to  foster  -a  better  understanding  of  the  weapons  acquisi¬ 
tion  process  and  better  communication  between  top  decision  makers  and 
Program  Managers.  The  OSD  should  conduct  a  symposium  for  the  managers,  of 
major  DOD  programs  where  ideas,  strategies  and  management  techniques  could 

s 

he  exchanged.  Each  Service  Secretary  might  consider  having  a  similar, 
symposium  for  his  service.  The  purpose  of  such  meetings  would  he  to  more 
closely  integrate  weapons  development  with  national  objectives  .and  to 
facilitate  the  exchange  of  successful  program  .management  techniques. 

Not  all  the  services  are  in  favor  of  establishing ^permanent  career 
fields  for  Program  Managers .  If  a  service  feels  that  line  officers  should 
he  "rotated  through"  a  program  office  rather  than  remain  in  thendevelop- 
ment  field,  it;  behooves  that  service  to  establish  effective  program  office 
organizations;;  Unfortunately,  organizations  then  tend  to  drive  the  PM-- 
and  those  which  are  in  existence  for  long  periods  of  time  stagnate  due  to 
lack,  of  dynamic  leadership.  Program  management  is  too  important  a ’function 
to  not  be  a  career  field  in  each  service. 

Continuity  in  the  assignments  of  good  Program  Managers  from  the 
"selling"  of  the  program  through  the  delivery  of  the  operational  systems 
to  the ’services  will  have  -a  pronounced  effect  on  .the  reduction  of  program 


risk.  The  COD  should  further  enhance  the  role  of  the  Program  Manager 

and  other  key 'personnel  in  the  development  career  field.  Reassignments- 

of  key  personnel  should  he.  made  only  after  major' program  milestones  have 

been  satisfied.  We  agree  strongly  With  the  Blue'  Ribbon  Panel  (lp) 

recommendation  to  increase  the-  authority  and  career  opportunities;  of  the 

Prograin  Manager  and  personnel.  These  key  personnel  should  he  identified 

for  tours  of  duty  of  sufficient  length  to  assure  program  continuity.  It 

may  he  appropriate,  for  instance*  to  assign  key  personnel  to  the  program 

through  the  next  major  milestone  with  options  for  longer  tenured  Whenever 

'?• 

possible,  these  key  personnel  should  overlap  assignments  with  their  re> 
placements,  so  that  a  .maximum* of  continuity  is  maintained . 

A  clear  system  of  rewards  and  penalties  must  be  established  that* 
provides  for  rapid  impact  on.  the  'Program  Manager's  career.  Diligence 
and,  above  all,  honesty  must  he  rapidly  rewarded;  inefficiency,  poor 
judgment  and  dishonesty  must  be  corrected  equally  as  rapidly.  But  more: 
than;:honesty  is  the,  issue,.  Program  Managers  are  in  a  crucial  position 
of  authority..  They  must,  impose  an  orderliness  of  their  own  design  oh 
the  program  so  that  the  process  (system)  does  not  drive  them.  PM’s.  that 
let  the  system  drive  them  will  soon; 'find,  themselves-  driven  into  cost 
overruns,  schedule  slippages  and  performance  degradations.  Such  lack  of 
management  ^ability  ,i's-  clearly  not  acceptable-.  One  of  the  best  ways  to 
make  a. change;  in- -the  Program  Manager's  career  is  to  give  him  visibility. 
High-level  visibility  should  be  provided  to'  the  managers  of  all  high 
priority,  programs.  For  instance;  periodic  briefings  of  high-level 
Service  officials,., such  as  a,  Service  Secretary;  should  be' made-  e:  part  of 
every  high  priority  program.,  If  intermediate  level  briefings  are 
desired  they  should  be  scheduled  back-to-back  with  the,  high-level 
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briefing  to  minimize  the  Program  Manager's  separation  from,  his  direct 
management;  function. 

Management  Under , Process.  Uncertainty 

For  the  purpose  of  discussing  useful  management  techniques  when  pro¬ 
cess  uncertainty  is'  anticipated  to  be  significants  the.  general  area  of 
process  uncertainty  has  been  divided  into  the  following  subgrouping: 
o  Uncertainty  of  the  criteria,  for  .program  approval 
o  Uncertainty  of  the  process  for  change  approval 
o  Uncertainty  of  scheduled  resources 
1.  Uncertainty  of  the  Criteria  for  Program  Approval.  Many  of 
thn-uncertainties  of  the  process'  surround  the  program  approval  process. 
Exactly  what  it  takes  to  get  through  the  D3ARC  and  move  into  the  next 
phase  of  the'  acquisition  cycle  is  unknown.  For  example*  current  policy 
directs  that  one  of  the  criteria  for  passage  must  be  a  completed  "risk 

•A 

analysis j"  yet  what  goes  into  such  a  document  or- what  would  cpnstitute 

'  >  \r  j  ti 

a  "good"  one  is  as  yet  unspecified  in,  ah  official  way..  A  part  of  '$j&s 
uncertainty  is  caused  by  the  BSARC  itself.  The  BSARC-  is  ridt  a  .body 
dedicated  to  a  single/purpose.  Rather .%  it  is  an  extra  duty  for  indi¬ 
viduals*  each  of  whom  spends  most  of  his  time  running  a  .large  defense 
staff  in  OSD,  In  the  absence  of  a  BSARC  staff*  'there  is  no-  way  to 
generate  d  real  "fealing"  for  the  BSARC 's  criteria  for  approval  without 

f  •  '  s, 

approaching  one  -of  the  high  level  appointees.  Thus  when  OSD  says  there' 
must  b/2  a  risk  -analysis  or.  each  program  before  it  is  allowed  to  pads  the 
DSARC,  there  4  s,  no  staff  to  f,6iiLow«-up  and  provide,  guidance  -to  'the  Program- 
Managers  on  the  preparation-  of  such  a  document.  The  assignment  to  DSARC 
of  a  small  staff  (perhaps,  from  within  BDR&E-  or  Systems  Analysis)'  night 
reduce  the  large  amount  of  uncertainty  surrounding  the  approval  criteria. 


The  DSARC  might  well  become  a  part.  of  the  Office  of  the  Deputy  Secretary 
of  Defense* 

2,  Uncertainty  of  the  Process  for  Change  Approval.  The  Program 
Mahager  is  the  central  figure  in  the  Weapon  Acquisition  Process;  he  stands 
iri  the  center,  among  the  forces  of  Congress,  OSD,  nis- 'Service  Headquarters 
Staff  ;  his  parent  Development  Command,  and  the.  using  Command.  The  rhetoric 
of  defense  management  says  that  he  is  the  individual  who  makes  tradeoffs 
among  cost,  schedule,  and  performance.  But  the  reality  of  the :matter  is 
that  he  has  severe  constraints,  imposed  by  each  of  the  above  mentioned 
organizations.  Except  for  relatively  minor  decisions,  it  is; unlikely 
that  the  Program  Manager  will  ever  make  a  significant  tradeoff  by  him¬ 


self.  Indeed-;  the  image  of  a  modern  Program  Manager  is  often  one  of  a 
traveling  salesman.,  flying  from  his  office-  to  the  contractor,  to  various 
headquarters,  or  searching  the  Pentagon  for  men  who  can  give  him  authority 
to  make  changes  ih  his  program. 


This  problem,  as  it  faces  the  Program  Manager,  is  one  of 
divided  authority  and  diffused  responsibility.  The  basis  for  having  a. 
multitude  of  staffs  and  a  diffuse  network  of  command  is  that  many  parts 
of  the  various  organizations,  need  inforasition.  on  the  new  system.  Yet 
this  need  is  seldom  balanced  by  a  benefit  in  meaningful  terms.  One 
alternative-  that  is  only  now  being  explored  is  limiting  the  number  .of 
individuals  that  can 'have  a'  major  voice  in  the  development  of  a  major 
program.  At  present  there  is  ho  clear-cut  appeal  level  for' tradeoffs; 
between-  the -.authority  granted  the  military  PM  and  the  'DSARC  level. 

Perhaps  it  is.  time  to  explore  the  possibility  of  a  Service  Steering  Group 
for  each  major  program.  This  group  would  be  composed  of  representatives 
from;  the  major -organizations  involved  in  the  approval  process  for 


program  changes.  It  could  consist  of.  delegates  from  the  Headquarters 

Staffs  the  Developing  Command  and  the  using  Command.  The  function  of  the 

Group  would  be  to  evaluate  and.apprcve  tradeoffs  in  the  program  on  a 

timely  basis.  'Ihe  managerial  level  of  the  Steering  Group  could  be„ 

tailored  to  the  priority  and  size  of  the  development  program,  being  perhaps 

general  officers  for  major  programs  .and  lower  ranking  officers  for  others. 

However,  it,  would  he  essential  that  the  representatives  he  delegated  the 

authority  to  make  tradeoff  decisions  for  their  organizations — not  just 

coordinate:  the  proposals.  The  absence  of  decision-making  authority  would 

mean  the  'group  would  become  merely  another  of  the  maiiy  obstacles  in  the 

path  of  the  Program  Manager.  The  scope  of  the  Steering  Group’s  authority 

•  • 

would  necessarily  have  to  be  limited  to  making  tradeoffs  within  the 
thresholds  established, by  the  DSARC .  Tradeoffs  beyond  the  DSARC  thresholds 
would  require  another  trip  up  the  DCP/DSARC  path. 

3.*  Uncertainty  of  the  Scheduled  Resources.  The  ^magnitude  of  this 
problem  can  scareiy  ,be  underrated'.  The  point  is  simply  that  the  number  of' 
agencies  that  can  tamper  with  the  scheduled  resources  (primarily  funding) 
of  an  individual  program  is  so  large  that  budget  uncertainty  is  almost 
always  the  single  largest  ^process  uncertainty.  Basically,  every  organi¬ 
zation  above,  the  Program  Manager  reserves  the  right  to  cut  the  program 

x 

funds.  The  potential  impact  of  this  uncertainty  can  be  illustrated  by 
an  example .  The  B-l  program  is  one  of  the  Air  Force’s  highest  (if  not 
the  .highest)  priorities.  Yet,  there  is  a  great  deal  of  uncertainty  about 
Its  getting  scheduled  funding  from  Congress.  The  impact,  in  terms  of 
total  system  cost  growth,  of  receiving  a  significant  cut  in  one  or  two 
.year’s  funding. is  potentially  greater  than, the  impact,  in  terms  of5 cost 
savings,  of -making >a  major  change  in  the  operational  retirements  (such 
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as  from  supersonic  to  subsonic).  The  required  stretch  out  of  the  program 
necessitating  parts  of  the  development  to  be  deferred  to  later  times 
would  increase  costs  by  rreduced  (development  optimization  and  by 
inflation. 

An  effective  technique  to  try  t  -  counter  drastic  -unexpected  oudget 
cuts  might  be-  the  establishment  of  a  sensitivity  analysis  with  frequent 
.updating  which  .would  provide  more  realistic  and  timely  information  on  the 
overall  cost  impact,  of  changes  in  ^scheduled  resources  when  cutbacks  are 
proposed.  Such,  information  might  have  a  deterring  effect,  on  the  approval 
of  resource  decrements  when  the  consequences  are  made  clear.  This  would 
mean  that  studies  on  the  effect 'Of  reduced  resources  would.have  to==be 
frequently  performed^  not  just  when-  the  need’  dictated.  .Generally,, 
sufficient  time  has  not  been  available'  in  the  past,  when  such  information 
has  been  requested-,  to  conduct  an  accurate,  analysis  . 
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CHAPTER  IV:  RISK  ASSESSMENT 
General  Considerations 

Risk  has’  previously  been:  described  as  -a-  leisure  of  a  particular 
aspect  or:  chapActeristici  of  a  proposed  weapon1  system  program.  jSpecif- 
Ically,  risk  is,  defined  as  the  probability' of  hoi  being'  able  to  acquire 
a  weapon  system  of  specified,  performance  characteristics  within  ,an 
allotted  time;,  under  a  given  cost  and  by  following  s  specific  course  of 
action.  A  formal,  risk, analysis  is  an  explicit  investigation  of  the  factors 
which’  affect  the  risk  associated1  with  a. program, and  a  written  presentation 
of  this  and  other  relevant  information  to  the  decision  maker.  A  risk 
assessment  is  a  comprehensive  and  carefully  structured  approach  for 
estimating  the,  risk  associated  with  a  particular  alternative.  Risk 
assessments  are  then  a  pert  of  risk,  analysis. 

Risk  assessment  is-  not  a  new  notion.  To  a  lesser  or  greater  degree, 
the  problems  of,  cost  growth,  schedule  slippage,  and  performance  degrada¬ 
tion-have  been  addressed  previously.  The  process,  however,  was  largely 
intuitive,  incomplete,  and  informal.  It  was  intuitive  in  that  a 
structured  quantitative  approach  often  gave  way  to  intuition  and 
"blackboard  analysis."  It  was  incomplete  in  that  detailed  analyses  of 
isolated  aspects  of  the  problem  were  rarely  brought  together  and  inte¬ 
grated  in  a  broader  analysis.  And  it  was  informal, In  that  the  results 
of  the  assessment  were -often  not  -written  and  explicitly  incorporated 
into  the  review/approval/control  process. 

The  greatest  potential  value  of  a  formal  risk  analysis  is  at  the 
■very  early  stages  of  the  concept  formulation  phase,  when  the  range  of 
possible  alternatives  is  greatest  and  the  really  substantial  decisions 
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have  yet  to  be  made.  Unfortunately,  because  of  a  lack,  of  both  quality 
and  quantity  of  input  data,  it  is  precisely  at  this  point  that  a  risk 
assessment  is  most  difficult  to  perform  and  the  output  most-  suspect. 

This  basic  dilemma*,  though  less  acute  in  lower  risk  situations,  will 
always  exist.  However,  significant  improvements  over  current  practices 
are  possible  immediately..  All  of  this'  is  not  to  suggest  that  &  risk 
analysis  at  the  early  stages  of  the  concept  formulation  phase  is  of  little 
or  no'  value.  Major  decisions  will  be  made  at  this  point,  and  they  will  be 
made  with  or  without  a  risk  assessment.  But  decisions  made  without  the 
benefit  of  such  an  assessment  will  be  made  in  the  absence  of  potentially 
valuable  information  available  from  no  other  source. 

Risk  changes  with  time,  and  may  even  increase  with  time.  Therefore, 
although  ah  early  initial  risk  assessment  is  necessary,  it  is  by  no  means 
sufficient.  A  risk  assessment  should  be  performed,  at  each  of  the;  major 
decision  points  in  the  development  of  the  program,  each  new  assessment 
using  the  previous;  assessment  as  a  j unping-off  point  and  improving'  and 
updating  it  to -reflect  changes.  Thus,  a  risk  assessment  should  be  per¬ 
formed  at  disc-'  te  points  in  the  development  process,  but  the  updating 
of  inputs  to  reflect  changes  is  'accomplished  continually.  These  points 
are  worth  stressing.  Risk,  assessments  are  useful  only  if  they  can -affect 
a  decision  to  be  made  among  various  alternatives.  -Needless  to  say,,  the 
assessments  must  also  be  a  quality  product. 

A  risk  assessment  of  good  quality  requires  the  'co-ordinated  efforts 
of.  a  highly  qualified,,  interdisciplinary  group,.  This  group  should 
consist  of  trained  analysts  in  mathematics,  probability,  statistics, 
operations  research,  and  computers  who  are  capable  of  putting  the  assess¬ 
ment  together,  cost  analysts  to  provide  cost  estimates,  design  and 
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engineering  people  to  provide  technical  performance  information,  and 
production,  people  to  provide  schedule  =and  integration  information. 
Additionally,  experts  in  the  various  technical  disciplines  must  be  avail¬ 
able  for  the  group's  use.  These  experts  supply  much  of  the  basic  input 
data  necessary  for  the  assessment,  wile  the  function  of  the, risk  assessment 
group  is  to  aggregate  these  inputs,  extract  meaningful  information  from  them, 
and  provide  this  information  to  the  decision  maker  in  an  understandable 
form. 

What  has  been  termed  a  risk  assessment  here  may  sound  to  many  like  a 
good  systems  analysis  approach  to  the  problem  of  risk.  And  so  it  is. 
Unfortunately,  once  that  connection  is  established  there  is  a  strong 
tendency  bn  the  part  of  many  to  transfer  all  the  grievances  associated 
with  poorly  done  systems  analyses  to  risk  assessment  and,  if  their 
aversion  to  systems  analysis  is  great  enough,  to  dismiss  it  entirely. 

This  attitude  is  unjustified,  though  understandable,  for  several 
reasons.  Aside  from  the  fact  that  some  poor  quality  work  has  been  done 
in  the  name  of  systems  analysis,  there  are  two  main  factors  which  con¬ 
tribute  to  this  situation.  C sc-  is  that  the  decision  maker  frequently 
does  not  understand  the  analyst  or  what  (he  does,  cannot  effectively 
communicate  with  him,  and  is  unable  to  evaluate  his  product.  On  the 
other  hand,  the  systems  analyst  often  tends  to  suffer,  from  a  lack  of 
perspective  and  fails  to  appreciate  the  fact  that,  as  important  as  it 
is,  his  output  is  only  one  of  the  inputs  which  the  decision  maker  must 
'act  upon*.  Additionally,  there  ,3?s  a  very  strong  inclination  on  the  part 

*s, 

of  the  analyst  to  re-define  a  problem  which  he  is  given  to  one  which  he 
knows  hew  to  solve-  by  techniques  he  is.  familiar  with. 

•These  considerations  suggest  the  need  for  still  another  member- of 


the  risk  assessment  group.  Namely,  an  individual  who  can  act  as  a 
liaison  between  tbe  decision  maker  and  the  risk  assessment  group— one 
who  can  understand  both  sides  and  effectively  communicate  between  them. 
Unless  there  is  such  a  built-in  provision  for  reducing  the  misunder¬ 
standings,  distrust,  and  antagonism  which  could  build  up  between 
management  and  the  assessment  group,  its  usefulness' to  the  decision  maker 
is  questionable.  The  assessment  group  and  the  decision  maker  must  work 
in  concert  if  the  output  of  the  group  is  not  to  become*  yet  another  required 
but  ignored  paper  study. 

As  has  been  previously  stressed,  the  results  of  risk  assessment  are  of 
value  only  if  they  can  affect  a  decision.  But  a  risk  assessment  is  just 
part  of  a  risk  analysis^  and  a  risk  analysis* is  just  one  of  the  many 
inputs  which  should  be  available  to  a  decision  maker.  It  is  nbt  a 
panacea,  and  it  should  definitely  not  become  a  master  cult  claiming  ability 
to  solve  all  the  problems  in  the  weapon  system  acquisition  process.  Oh 
the  other  hand  the  contribution  of  a  good  risk  assessment  in  this  area 
can  be  both  unique  and  substantial. 

Sven  in  those  situations  where  the  quality  of  the  risk  assessment  is 
restricted  by  scarce  cr  poor  input  car a,  such  as  at  the  early  stages 
of  the  concept  formulation  phase,  the  structured  investigation 'and 
inquiry  which  precede  the  actual  assessment  can  be  of  great  value  in 
calling  attention  to  potential  problem,  areas'.  As  more  resources  are 
devoted  to  the  assessment  and  more  data  becomes  available  to  work  upon, 
the  better  will  be  the  results  of  the  assessment. 

The  results  are  not  without  cost,  however,  and  the  level  of  effort 
has  to  be  commensurate  with  the  Value  of  the  program.  A  massive  risk 
assessment  on  either  a  relatively  minor  program -or  on -one  which  does 


not  represent  a  state-of-the-art  advance  is  unwarranted.  On.  the  other 
hand, "against  the  background  of  staggering  cost  overruns  of  most  major 
weapons  systems,  a  good  risk  assessment  which  can  help  reduce  these 
excessive  costs  will  pay  for  itself  many  times  oyer; 

Quantitative  Disciplines  Involved 

In  this  section,  a  very  brief  treatment  of  certain  of  the  more 
significant  disciplines  involved  in  quantitative  srisk  assessment  is  given. 
It  should  be  noted  that  each  of  the  areas  mentioned  has  an  extensive 
literature  .{a  part  of  which,  is  referenced  in  the  Bibliography)  which  is 
impossible  to  summarize  in  just  a  few  pages.  Nevertheless,  an  attempt 
will  be  Side  to  give  the  reader  a  nodding  acquaintance  with  each  subject 
and  to  highlight  the  advantages  and  limitations  in  its  use. 

Subjective  -Probability 

In  the  development  of  a  weapon  system,  (or  indeed  any  new  product)  it 
is  impossible  at  the  outset  to  know  with  certainty  what  the  final  outcome 
will  be  in  terms  of  completion  time,  cost,  and  performance.  At  the  same 
time,  decision  makers  end  technical  experts  are  not  completely  ignorant  of 
what  the  outcomes  may  be.  Their  state  of  knowledge  is  somewhere  between 
these  two  extremes,  and  it  is  useful  bp-  have  a  language  which  will 
express  their  degree  of:  belief  that  certain  outcomes  will  occur.  Sub¬ 
jective  probability  is  such  a  language-.. 

By  convention,  a  subjective  probability  of  1.0  is  assigned  to  an 
event  which  the  assessor  is  sure  will  happen  and  a  probability  of  0.0 
is  assigned  to  an -.event  which  he  is  sure  will  hot  happen.,  For  events 
whose  occurrence:  i-s  uncertain  in  the  ’sdind  of  the  assessor,  -a  number 
'between  0.0  and  1.0  is  assigned,,  which  reflects  his  degree  of  belief 


that  the  event  will  happen.  The  only  additional  restriction  on  the 
assignment  of  subjective*  probabilities  is  that  if  a  collection  of  events 
are  mutually  exclusive  (not.  more  than  one  can  occur  at  the  same  time) 
and  exhaustive  (one  of  the  events  must  occur)  then  the  sum  of  the.  pro¬ 
babilities  of  the  individual  events  mist  be  1.0. 

With  these  conditions,  "subjective  probabilists"  claim  that  the 
calculus  cf  probability  follows  in  the  same  way  as  in  the  sore  usual 
objective  theory  of  probability,  in  which  the  probability  of  ah  event 
may  be  interpreted  as  the  expected  relative  frequency  with  which  it 
occurs  in  repeated  experiments .  While  acknowledging  that  it  is  possible 
to  express  degree  of  belief  as  a  number  between  0.0  and  1.0*  the  "object¬ 
ive  probabiiist"  has  serious  reservations  about  the  reasonableness  of 
any  further  calculation  with  such  numbers.  While  this1,  controversy 
doubtless  will  not  be  resolved  in  the  foreseeable  future,  the  fact  remains 
that  decision  makers  must  still  make  decisions,  and  subjective  probability 
has  been  shown  to  be  a  useful  tool  in  dealing  with  uncertainty  in  &  quanti¬ 
tative  way. 

Several  methods  have  been  developed  for  elieitng  an  individual's 
subjective  probability  of  the  occurrence  of  an  event,  and  research  in 
this  area  is  continuing.  These  methods  .Sar§  geared  to  helping  the 
individual  portray  in  probabilistic  terms  what  his  beliefs  are.  They 
do  not  come  to  grips  with  the  problem  of  to  what  extent,  hia:  beliefs 
reflect  reality,,  a  limitation  which  should  always  be  kept  in  mind. 

It  should  also  be  noted  that  -a  subjective  probability  assigned-  to 
the  occurrence  of  a  single  event  cannot  in  any  way  be  "validated";  so  \ 

any  attempt  to  do-  so  is  draught  -with  failure .  -The  event  will  either 
occur  or  not  occur,  and  will  not  occur  with  a  certain  probability.  To 
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overcome  this  difficulty,  e  "theory  of  scoring  rules"  is  being  developed, 
by  -which  the  assessing  capability  of  experts  can  be  measure^.  Although  this 
theory  is  in  its  infancy,  it  could  be  very  useful  in  the  if  the 

trend  toward  the  use  of  subjective  probability  distributions  for  uncertain 
events  continues; 

Trend  Extrapolation 

Because  of  the  long  leadtimes  typical  in  acquiring  new  weapons 
systems,  it  is  important  to  have  some  way  of  estinating  what  level  of 
technology  may  be  attainable  some  years  hence.  This  need  is  to  some 
extent  satisfied  by  the  use  of  trend  extrapolation,  a  quantitative 
abroach  for,  predicting  future  technology.  •* 

Performance 

parameter 


Figure  5. 

Trend  Extrapolation 


The  points  on  the  diagram  above  represent  specific  technological 
achievements  in  the  past  and1  a  curve  (in  this  case  a  straight  line)  is, 
fit  to  the  points .  Then,  by -extrapolation* along  the  curve,  one  predicts 
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the  level  of  performance  that  is  reasonable  toexpeet  at  some  future 
data.  In  doing  this,  one  assumes  that  the  established  trend  will  con¬ 
tinue  ana  that  the  present  is  not  a  point,  of  discontinuity  or  a  break 
point  in  the  curve v 

Boring  the  early  conceptual  stages/  of  an  advanced  weapons  system, 
trend  extrapolation  is  a  useful  tool  by  which  the  planner -may  project 
the  technical  capabilities  of  his  product.  As  the  design  becomes  more 
certain  it  can  play  an  important  role  in  projecting  the  feasibility  of 
subsystems  and  components1. 

It  also  serves  to  "real  flag"  proposals  which  would  appear  to  be 
beyond  the  projected  state-of-the-art .  This. 'is  not  \to  say  that  any 
projected  performance  parameter  which'  lies  above  the,  curve  cannot  be 
achieved,  but  rather  that  such  items  should  be  closely  watched.  By  the 
seune  token,  projected  parameters  which  are  on  or  below  pile  curve  are  not 
necessarily  easily  a'chieveable.  Unfortunately,  although1  trend  extra¬ 
polation  may  indicate  that  a  certain  technology  is  reasonable  to 
expect,  it  provides  no  information  cn  how  to  go.  about  achieving  that 
adyanced  technology. 

Group  Assessment 

Frequently  in  estimating  cost,  schedules,  and  performance  during 

the  development  of  a  new  weapon  system,  a  particular  question  may  be,  so 

/* 

complex  or  so  important  that  a  group  of  experts  is  asked  to  -.consider 
the  ;problem,  under  the  assumption  that  "many  heads  are  better  than  one." 
Although  history  is  replete  with  counter-examples ,  in  most  cases  this 
assumption  is  probably  valid. 

The  techniques  of  group  assessment  may  be  classified  by  the  amount 


of  information  each  expert  has  shout  the  estimates  of  the  others. 

On  one  extreme,  each  e.peri  may  he  asked  to  estimate  t 
prccien  co|h2.c-;.<.^y  inaeper.de r.fiy.  of  the  others.  Then  a  weighted 
average  approach  is  used  to  combine  the  individual  answers  into  a 
single  "group  estimate „ "  Although  this  method  has  the  advantage  of 
not  subjecting  the  Individuals  to  intimidation  by  others,  two  obvious 
disadvantages  are  the  fact  that,  the  individuals  cannot  take  advantage 
of  the  knowledge  of  the  others  and  the  difficulty  in  formulating  a 
meaningful  weighting  scheme. 


On  the  other  extreme  is  direct  face-to-face  confrontation  of  the 
experts  with  the  requirement  that  a  group  consensus  be  reached.  Here 
a  weighting  scheme  is  unnecessary  and  full  exchange  of  information  is 
possible,  hut  significant  problems  of  intimidation  may  occur. 

Although  there  -sure'  others,  the  DELPHI  technique  has  been  proposed 
as  a  middle  ground  between  the  above  extreme  alternatives .  .DELFHI  does 
not  permit  face-to-face  discussions ,  but  rather  has  each  expert  .submit 
anonymously  the  required  estimate  together  with  a  written  justification 
for  it.  ‘These  statements  are  then  circulated  to  all  the  other  experts 
and  each  is  again  asked  ‘to  submit  another  estimate  together  with  a 
justification.  The  process  continues  until  little  change  in,  the 
estimates  from  round  tc  round  occurs .  Then  the  median  value  is,  chosen 
as  the  group  assessment.. 

!r.  numerous  experiments  conducted  by  RAND -with  the  DELFHI.  technique, 
it  has  been  observed  that  after  several  rounds  the  estimates  tend  to 
•converge  on  a  particular  value,  though  not  necessarily  on. the  "correct" 
value,  of  course. 
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Research  in  the  area  o?  group  assessment  is  continuing  with  en^hasis 
on  methods  for  stripping  away  onuanted  psychological  factors,  and  deter¬ 
mining  the  optimal  amount  and  type  of  information  exchange  amorg  the 
experts . 

Cost  Estimating 

Although  thire  are  many  specialized,  techniques  used  by  cost 
estimators,  the  basic  method  is  to  break  the  project  down;  Into  sub¬ 
systems  and  estimate  the  cost  of  each  by  extrapolating  the  cost  of  similar 
previously  developed  systems .  This  approach  works  reasonably  well  pro¬ 
vided  the  subsystem  to  be  estimated  is  not  radically  different  from 
previous  ones.  When  it  is  (and  this  is  becoming,  increasingly,  common), 
the  cost  estimates  are  at  best  merely  educated  guesses. 

Cost  estimating  methodology  has  already  progressed  far  beyond  the 
available  data  base.  Due  to  non-uniform  procedures  and  inadequate  book¬ 
keeping  in  tile  past,,  it  is  extremely  difficult  to  ascertain  the  cost  of 
•existing  systems.  Without  these  data,  it  is  hard  indeed  to  make  reli¬ 
able  estimates  of  the  cost  of  future  systems. 

Another- 'significant,  problem  inherent  in  the  process  is  the 
difficulty  irv  estimating  the  cost  of  integrating  the  subsystems .  Each 
integration  problem  appears  to  be  quite  different  from  previous  ones, 
so  the  extrapolation  technique  loses  some  of  its  validity; 

These  difficulties,  are  mitigated  to  some  degree  by  having  engineers 
as,  a>;part  of  the  cost  estimating  team,  a  practice  which  appears  to  be 
gaining  .-ground  . 

A  common  complaint  among  cost:  estimators  in  the  'aerospace  industry 
.is  that  while  they  .usually  have  sufficient  time  -to  make  the  initial 
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estimate,  frequently  they  are  required  tc  "cost  out"  changes  -to  the 
design  in  unreasonably  short  periods  of  time,  thereby  inducing  errors 
that  would’ not  otherwise  be -present. 

All  too  frequently  there  appears,  to  be  insufficient  communication 
between  cost  analysts  arid  the  designers,  system  analysts,  arid  the  manage¬ 
ment,  structure.  More  communication  would  undoubtedly  result  in  timely 
deoigri  tradeoffs  that  would  not  impair  the  performance  of  the  product, 

and  would  bring  it  in  at  reduced  cost  .. 

6  ■  -  .  ■ 

Finally,  even  though  single-point  estimates  are  required  for  book¬ 
keeping  purposes,  cost  analysts  must  be  trained  to  give  estimates  in 
terms  of  subjective  probability  distributions  for  use  as  inputs  to 
quantitative  risk  assessments,. 

Network  Analysis 

At  the  point  in  the  development  of  a  weapons  system  where  the* 
project  is  sufficiently  well-defined  thau  the  various  subtasks  may  be 
sequenced  and  milestones  established^  it  is  possible  to  use  a  network 
as  a  mathematical  model.  In  tuis  netware  the  milestones  are  represented  by 
nodes,  the  activities  by  branches,  and  time  and  post  estimates  for 
activities  by  probability  distributions  assigned  to  the  branches.  By 
using  any  one:  of  a  number  of  techniques,  it  is  then  possible  to  obtain 
probability  distributions,  for  thV  time  and  cost  of  the  entire  project 
and  the  probability  that  it  will  Toe  successfully  completed,.  These 
data,  then,  can  be  easily  used  to  obtain  the  risk  of  completion  within 
specified  time  and  cost  constraints. 

Several  types  of  networks  to  represent  a  development  project,  have 
been  proposed  and,  are  in  use.  Prpbably  the  most  widely  known  is  PERT. 


While  PERT  may  he:  of  value  in  planning  a  project  in  which  there  are  few 
uncertainties ,  it  is  generally  unsuitable  for  developing- new  weapon 
systems,  the  two  most.  significant  reasons  for  this  unsuitability  are 
the  fact  that  the  only  permissible  nodes  are  AND  nodes  (which  means  that 
there  can  be  no  probabilistic  branching),  and  that  critical  paths  are 
computed  on  the  basis  of  their  expected  values  (which  means  that  paths 
that  appear  to  be  non- critical  may,  in  fact  be  the  potential  bottlenecks). 
A  lesser  known:  technique ,  Critical  Path  Method  (CPM)  suffers  the  same 
drawbacks. 

GERT  is  .a  -network  technique  which  overcomes  some  of  the  difficulties 
cf  PERT  in  that  it  admits  probabilistic  branching  but  it  suffers  from  a 
restricted  number  of,  types  of  nodes  that  may  bs  used  (in  particular,  AND 
nodes  are  difficult  to  deal  with) . 

Network  simulation  appears  to  be  the  best,  technique  available  at  the- 
present  time ;  The  types  cf  nodes,  and  distributions  on  the  branches 
which  can  be  used  are  limited  only  by  the  level  of  the  computer  pro¬ 
gramming  effort  involved.  A  number  of  large  scale  simulation  programs 
are  available  which  would  be  suitable  for  use  in  a  risk  assessment  of 
-a  development  project. 

Quantitative  vs  'Qualitative.,, Risk  -Assessment 

The  difference  between-  a  quantitative  and' -a -qualitative'  risk;  .assess? 
meat  is,  clear.  A  quantitative,  risk  .assessment  is.  one  in  which,  thee 
estimate  of  the  risk -associated  with  a  program  is  expressed  as  a  . number. 
A  qualitative  risk  assessment  is- one- in  which  the  estimate  fpf  the  risk 
associated  with  a  program  is  expressed  in  non- numeric,  terms;  Much  as' 
high y  medium,  or  low.  With  the  exception  of  the  group1  assessment 
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techniques,  the,  disciplines  which  were  discussed  previously  are  all 

| 

quantitative  Methods.  Qualitative  '’methods,''  on  the  other  hand,  involve 
fairly  simple  and  direct  ideas  such  as  ordinal  rankings,  etc. 

There  is  no  doubt  that  when  both  can  be  used,  quantitative  risk 
assessment  is  infinitely  more  desirable  and  useful..  However,;  there  are 
times  when  the  numerical  data  which  are  required  for  a  quantitative 
assessment  cannot  be  obtained,  and  in  these  cases  there  is  no, choice  but 
to  employ  a  qualitative  approach..  In  fact,  at  the  very  early  stages  of 
the  acquisition  cycle  a  qualitative  approach  may  be  the  only  one  possible. 
The  great  disadvantage  is  that,  unlike  quantitative;  methods,  it  is  very 
difficult  to  aggregate j  combine,  and  compare  qualitative  information. 
However,  it  is  a  mistake  to  assign  really  arbitrary  numbers  simply,  to 
permit  one  to  bring  quantitative  methods  to  bear. 

Summary  and  Recommendation. 

The  group -assessment  techniques  and  the  disciplines  of  subjective 
probability,  trend  extrapolation,  and  cost  estimatings  can  be  used  to, 
produce  terminal  results  or  to  generate  inputs  to,  the  more  powerful 
scheme  of  network  analysis.  The  outputs  which  they  produce,,  though 
useful,  do  not  supply  the  kind  of  information  necessary  for  a  quanti¬ 
tative  risk  assessment  c-f  the  type  required.  The  technique  which 
offers  the  most  promise,  in  quantitative  risk  assessment  is  a  versatile, 
simulated  network  approach  using  group  assessment  techniques t  sub¬ 
jective  probability technological  forecasting-,  cost  estimating,  ^and 
others  as  sources  of  input., 
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Network  Simulation  in'  Risk  Assessment  -v.-An  Ex 


It  is  very  difficult  for  someone  who  is  not  familiar  with  network 
simulation  to  understand  how  it  can  supply  information  useful  in  a  risk 
assessment.  The  following  simple ,  but  illustrative  example,  is  furnished 
to. provide  some  insight  into  the  ‘process  and  confidence  in  it. 


Suppose  a  system  consists!  of' only  two  components,  subsystem  X  and 
subsystem  Y,  and  that  the  course,  of  action  to  be  easployed  iii  building 
the  system  is  to  produce  each  of  the  subsystems  at  the  .same  time.' and 
then-  integrate  them.  The  essential  features  .of  this  simple  ^process 


can  be  abstracted  and  modeled  by  the  following  network,  where  the 
nodes  represent  the  milestones  and- the  branches -represent  the  activities 


as  indicated. 


Production  on 
subsystem  X 
begun 


subsystem  Y 

begun  Figure  6. 

Network  ‘Representation,  of  a;  Simple  .System- -Acquisition 


Now,  although  experts,  familiar  with  the  technology  required. to 
produce  a  subsystem  will  typically  be  unable  to  predict  precisely  how 
long  it  will  take  or  how  much  it  will  cost  to  build  it ,  they  will  have 
some  idea  of  what  values  are  possible  arid  which  are  more  likely.  This 
information  is  elicited  from  them  in  the  form  of  a  subjective  probability 
distribution. 

For  simplicity*  suppose  that  a  probability  distribution  of  the  time, 
tjf  ,  which  is  expected  to  be'  required  to  produce  subsystem  X  is  elicited 
from  experts  on  subsystem  X.  Also,  suppose  that  a  probability  distri¬ 
bution  of  the  time,  ty  ,  which  is  expected  to  be  .required  to  produce 
subsystem  Y  is  elicited  from  experts  on  subsystem  Y’.  Finally, 
suppose  that  a  probability  distribution  of  the  time,  ty  ,  which  is 
expected  to  be  required  to  integrate  the  two  subsystems  is  elicited 
from  integration  experts. 

Additionally,  suppose  that  cost  estimators  indicate  that  all.  costs 
are  linear  functions  of  the  time  involved.  In  other  words,  the  cost, 

Cyr  ,,  to  build  subsystem  X  is  Cjr  =  a^  +  b1  ty  ;  the  cost,  Cy  .  to 
build  subsystem  Y  is  Cy  =  8g  +  bgty  ;  and-  the  cost,  6y  ,  to  • 
integrate  the  two  subsystems  Is  c_  =  +  b'^ty  .  In  these  equations 

the  constants  a-j.  ,  ag  ,  represent  the  fixed  costs  while  the 

■constants  b_  ,  bk  ,  b'  represent  the  time -variable  costs  for  the 

r~  ^  '  i’ 

prospective  activities,  and  a  specific  value  for  each  of  them  would 
-be  f irnished.  by  the  cost  estimators. 

An  example  distribution  of  time  and  the  cost  equation  associated 
with  each  of  the  three  activities  (branches),  in  the  plan  of ^action  , 
(network)'  employed  in  the  building  of  the  system  are  displayed  in  the 
figure  below.; 


75 


The  question  at'  this  point  is  this:  Given  the  information  above,  what 
.is  the  total  system  expected  to  cost  and  how  long  is  it  expected'  to 
take  to  build?  The  answer  is  provided  by  simulating  the! network, 
which  is  accomplished  in  the  following  manner. 

Using  a  computer  and  random  numbers, ..a  representative  production 
time  for  subsystem  X  is  generated  from  the  distribution  of  times 
given  by  the  experts  and  the  corresponding  cost  of  building,  subsystem 
X  is  computed  from  the  cost  -equation  furnished  by  the  cost  estimators. 

7? 


t 
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The  same  thing  is  done  to  generate  a  time  and:  cost  for  rbj^lfiing;  subsystem 
Y  .and  for  integrating  the  two..  ‘  .  -  _ 

P*or  this  first  pass  through  the  network,  the  cost*  §  of  building,  ' 
the  total,  system  is  simply  the  sum  of  the  three  costs  computed  idtfre 
paragraph  above;  The  time,  T  ,  required  to  build  the  total  system  is 
simply  the.  longer  df  ;the  times  it  took  to  build  subsystems  X  and  Y 
plus  the  time  it  took  to  integrate  them.  We  have  now  simulated-  the 
building  of  the  system  in  the  computer  one  time  and  have  obtained!  a 
total  time,  T^  ,  and  a  total  cost,  CL,  .  — 

Another  pass  is  next  made  through  t-hei  network  to  simulate  the  building; 
of  the  system  a  second’  time.  Since  the,  times  selected  in  this  .pass  will 
not  he.  the  same  as  before,  a.  different  total  time,  Tg.  ,  and  a  different 
total  cost,  Cg  ,  will  be  obtained.  We  now  have  two  pairs  of  time  and 
-cost. 

Depending  on  the  size  and  complexity  of  the  network,  hundreds  or 
thousands  of,  additional  passes  through  the  network  will  then  be  made . 

This  is  done  to  give  the  "laws  of  chance"  sufficient  opportunity  to 
work  so  that  the  results  Will  be  representative  and  in  the  correct 
proportions. 

This-  whole  network  simulation  process  yields,  among  other  things, 
pairs  of  total  time  and  cost  for  the  system.  They  may  be  displayed  in 
a  table  as  follows;. 


Simulated  Output 
Total  Time  .  I  Total  Cost 
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Table  3 

Simulated  Distribution  of  Time  and  Cost; 


where  ri  is  the  total  number  of  parses  made  through  the  network  during 
the  simulation. 

The  column  of  costs,  in  the  table  above  is  then  used  as  the  estimate _ 
of  the  distribution  of  the  total  cost  of  the  system.  And  from  this  the 
risk  profile  of.  cost  can  be  constructed.  Its  graphical  presentation 
might  appear  as  follows:. 

Risk 
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Total  system  budget 


Figure  8. 

Risk  Profile  of  Cost 


In  the  figure  above,  the  risk  associated  with  a  budget  allocation  of 
isi  seen  to  be  about  0.4?.  That,  is,  the  probability  is  about  0.45 
that  the  total  cost  of  building  the  system,  will  be  wore  than  $CQ  . 

Where  did  this  come,  frois?  It  came  from  eh  examination,  of  the  co!\ian  of 
simulated  costs,  in  the  table  above- -an  examination  which  in  this  case 
would  have  shown  that  about  45$  of  the  n  s initiated  costs' were  greater 
than  $CQ  .  -  - 

This  same  technique  is  used  to  construct  the  rest  of  the  risk  profile 
of  cost,  and,  using  the  column  of  simulated  times,  to  construct  the  risk 
profile  of  time.  A  similar  technique  can  be  used  to  construct  the  joint 
risk  profile  of  time  and  cost.  And  increased  computer  programming 
effort  and  different  techniques  can" yield  yet  additional  valuable 
information  on  the  risk ^  bf-abuiiding=the  ^ystekr 

Information  of  this  now  be  used,  to  make  decisions  con¬ 

cerning  cost,  schedule,  perfcr^lHcpj,.  and  aqqMsitiqh  strategies.  It 
can  be  very  useful  in  establishing  realistic  program  budgets  and 
schedules  or  in.  modifying  existing  ones.  Additionally,,  simulations  of 
alternate  plans  of  action  for  building  a  system  provide  an  excellent 
means  of  comparison  among,  them  on  the  basis  of  risk  considerations. 


CHAPTER  V:  RISK  ANALYSIS 

■The  Nature  of  a  Formal  Risk  Analysis 

It  nay  seem  that  a  natural  result  of  a  study  of  risk  analysis  would 
be  a  very  detailed  methodology  or  technical  manual  for  actually  per¬ 
forming  a  risk  analysis  of  a  proposed  weapon  system  and  a  specific  format 
for  presenting  the  results.  However,  because  of  the  great  differences  in 
various  projects  and  the  time  span  from  the  early  conceptual  phase  to  the 
la\,i  development .phase  in  which  risk  analyses  should  be  performed,  no 
single, style  of  analysis  is  universally  applicable..  Indeed,  it  is  hard 
•to  find. a  single  style  of  analysis  which  is  even  representative.  On  the 
other  hand.,  it  would  be  equally  inappropriate  for  this  study  to  be  no 
more  specific;  than  to  merely  exhort  planners  to  perform  some  ill-defined 
risk,  analysis  . 

Therefore,  we  have  chosen  a  middle  ground'  ir.  the  sense  that  we  will 
identify  the  general  nature  of  a  risk  analysis  and  specify  the  outputs 
that,  should  result  from  a  good  risk  analysis,  of  virtually  any  type  of 
weapon  system^ at  almost  any.  stage  in  its  development. 

Risk  analysis  is  a  highly  coordinated  examination  of  all  factors 
which .affect  the  risk  of  acquiring  a  weapon  system.  The'  coordination 
which  is  .necessary  is  between  two1  different  . areas .  First,,  there  are 
the  various  strategies  which  are  available  for  dealing  with  the.  problems, 
which. arise  from  uncertainty!  i.e.,  the  risk  management;  strategies. 

These  were  treated  in  Chapter  III.  Secondly,  there  is  the  estimation 
(either  quantitative  or1  qualitative-)  of  what  the’  risk  associated  with 
a  specific  course  of  action/, actually  is,  i.e,.j-  .-xvisk  assessment.  This* 
was  treated  in  Chapter  IV.  In  a  normative  fashion,  we  , shall  now  clarify 
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the  nature  of  the-  coordination  "between  these  two  areas  that  is  required' 
for  a  risk  analysis. 

1.  We  begin  at  that  point  in  time  when  a  particular  -program 
is  proposed  in  order  to  satisfy  an  alleged  need.  The  first  step,  in  a 
risk  analysis  is  to  assess,,  either  quantitatively  or  qualitatively,  the 
risk  associated  with-  the  program  as  it  is-  proposed.  This  is  accomplished 
by  the  risk  assessment  group  using  the  techniques  introduced  in  Chapter. 
IV,  and  it  does  hot  rely  on  the  generation  of  management  alternatives. 

2;  The  second  step  in  a  risk  analysis  io?  to  identify  those 
aspects  of  the  proposed  program,  which  are  considered  to  be  problem  areas1. 
This  is  a  coordinated  effort  between  the  manager  and  the  risk  -assessment 
group.  The  manager,  with  the  aid  of  his  stuff  and  specialists,  deter¬ 
mines  those  aspects  of  the;  proposed- program  which. axe  potential  problem 
areas  from  a  technical,  budgetary,  production  or  design  viewpoint.  The 
risk  assessment  group  determines  those  aspects  of  the  proposed  program 


which  are  potential  problem  areas  from  a  sensitivity  analysis,  provided 


the  assessment  is,  quantitative.. 


-A  sensitivity  analysis  is  an  examina¬ 


tion  of  how  sensitive  the.  results  of  the  assessment  (the  ;risk  associated 
with  the  program)  are  to  .changes  in  the  inputs »  It  -reveals  which 
factors,  when  altered,  cause  significant  changes  in  the  results.  It 
also  reveals  which  of  the  Input  factors  can  be  altered  and  still  not 
significantly  affect  'the  results  of  the- (assessment. 


-The  manager  and  the  risk  assessment-  "group-  now.  need  to 


combine  the  results  of  their  separate  efforts  and  arrive  at  a  single 
set  of  potential  problem  areas.  Obviously,  the  set  will  contain  all 
of  those)  areas  which  both1  the  manager  and  risk  assessment  group  singled 


out  as  being  potentially  problematical,  and  it  will  not  contain  any 
of  those  areas  which  both  singled.,  out  as  non-problem  areas.  However, 
it  is  not  necessarily  true  that  the  cet  should  contain  all  of' those 
areas  about  which  the  manager  and  the  risk  assessment .group  -disagree. 
There  my  be  areas  which  the  manager  singles  out  as  .potential  problem 
areas  which  can  be  shown  to  have  little  affect  on  the,  overall  risk  oh 
the  basis  of  .a  sensitivity  analysis .  These  should.be  excluded  from,  the 
final,  set.  (Similarly,  there  maybe  areas  which  the  risk  assessment 
group  singles  out  as  potential  problem  areas  which  the  manager  is  con¬ 
fident- will  not  change.  These  too  should  be  excluded  from  the  final, 
set.  The  need  and  benefits  of  coordination  in  this  step  are  clear.. 

3.  Now  that  those  areas  whose  impact  on  the  risk  of  the  pro¬ 
gram  is  , potentially  adverse  have  been  isolated,  the  third  step  in  a 
risk  analysis  is  to  propose',  means  of.  avoiding  or,  reducing  these  problem': 
areas-.  This  task  falls  largely  to  the  manager  and  his  staff  and 
specialists  and  it  is  accomplished  by-  the  incorporation  of  appropriate 
acquisition  strategies  discussed  iii-Chapter  III.  The  result  of  this 
step  Is  the  generation  of  new,  and  presumably  better.,  alternative 
courses  of  action. 

4.  At  this,;  :poiht  we  are  ready  to  repeat  the  above  sequence 
beginning  with  step  one.'  for  each  of  these  new  alternatives.  This 
iterative  process  is.  then  continued  until  nc.  basically  new  alternatives 
are  proposed.  This,  then,,  is  the  nature  of  a  risk  analysis.  The 
formalization  of  the  analysis- is  accomplished  by  the  format  ingand 
presentation  of  the  results  to;  a  decision  maker.  We  now  turn  our 
attention  to  a  normative  discussion  of  what  the  outputs  of  the  risk 
analysis  should  be. 
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Outputs  of  a  Formal  Risk  Analysis 

1,  A  -generaldescriptidn  of  the  dominent  uncertainties  (target-, 
technical;  internal  program,  or  process)  which  directed"  the  selection. 
of  the  original  course  of  action. 

2 .  Identification  of  alternate  courses  of  action  (such  as- hardware 
proofing >  parallel  development ,  etc , )  to  resolve  the. major  uncertainties'. 

3,  'A  detailed  discussion  of  the  potential  problems  in  each,  major- 
program  element -for,  each  course'  of  action  .considered.  In  addition  to 
technical  uncertainties,  this  discussion  should  include  process 
uncertainties  (e.g.,  budget),  target  uncertainties  (e.g.,  engineering 
specifications)  and  internal  program  uncertainties  (e.g.,  key  personnel). 

4.  Individual  and  joint  risk  profiles  of  time  and  cost  for  each 
alternative  course  of  action.  The  Inherent  assumption  is  that  the 
specific  desired  performance  is,,  obtained  by  following  the  course-  of 
action. 

One  of  the  principle,  outputs  of  the  simulation  of  a  network  in 
which  the  tiAe  and  dost  associated  with  each  branch  is  a  random  variable, 
is  -a.  joint  distribution  of  time  and' cost  for  the  entire  project.  And 
although,  it  can  he  displayed  in  a  three  dimensional  figure,  this  joint 

'.distribution,  is  difficult  tp.work  with'.  However,  since  time  and  cost 

% 

are  dependent,  it  is  essential  to  be  able'  to  see  how  they  combine  to 
affect  the  risk  of  the  program.  This  can  be  accomplished  with  the  aid 
of  a  joint  risk  profile,  of  time  and  cost  (which  is  easily  obtainable 
from  the.  joint  distribution  of  time  and  cost)  similar  to  that  shown 


below. 


Cost 


Joint  Risk  Profile  of  Time  andCost 


The  curves’  on  this  graph  are  lines  of  equal  risk.  For  a 
particular  schedule  and  a  particular  budget,  the  total;  program  risk 
can  be  approximated  by  interpolating  between  the  curves.  For  example- , 
the  total  program  risk  associated  with  a  budget  of  c  and  a  schedule 

.  4  -  -O  • 

of  tg,  is  about  0.25;,  i.e.,  the  probability  that  the  program  will 
cost  more  than:  :and  take  longer  than  tq.  is  about  0;25. 

Also  obtainable’  from  the  output  of  the  simulation  of  a  network 
in  which;  the  time  arid  cost  associated  with  each  branch  (activity)  is 
a  random  variable  is  the  individual  distribution  of  total  program, 
cost  and  the  individual  distribution;  of  total,  program  time.  And. 
from  these  , are  easily  obtained  the  individual  risk  profile  of  total 
program  cost  and  the  individual  risk  profile- of  total  program  time. 
Typical  graphs  are  shown  below. 
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Figure  10 i 

Risk  Profiles  of  'Time:*  Cost 


For  example,  if  the  total  time  allotted  for  the  project  is  tQ 
weeks,  then  the  risk  associated  with  this  schedule  is  r^  ;  i.e;,  the 
probability  that  the  project  will  take  longer-  than  tQ  weeks  to  complete 
is,  rt  t  Similarly,  if  cQ  dollars  are  budgeted  for  the  project- ‘the 
risk  associated  with  this  funding  level  is  r  j  i.e.,  the  probability 
that-  the  program  will  cost  more  than  $c0  is  r  Mote  that  since 
time  and  cost  are  not  independent,  the  probability  that  the  program 
will  cost  more  than  $c  and  take  longer  than,  t  is  not  r  r.  — it 
can  only  be  obtained  from  the  joint  risk  profile  described  above. 

8 .  An,  analysis  of  how^.sensitive  the  risk  profiles  are  to  change  in 
the  [Inputs.  It  is  crucial  to  .know  which  factors,  when  altered,  -cause 
significant  changes  in  the  risk  profiles.  It  is  also  essential  to -know 
which  of  the  input  factors  can  change  and  still  not  significantly  affect 
the  risk  profiles. 
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7.  Comparison  with  previous  risk. 'analyses  of  the  same  project.  For 
example,  if  the  risk  profile  of  cost  changed  from  curve  A  to  curve  B,  it 
would  indicate  the  risk  associated  with  cost  had  increased. 


Comparative  Risk  Profile  ok  Cost, 


8.  A  Comparison  of  the  candidate  management  courses  of  action  and  a 
recommendation  of  a  preferred  course,  of  action  on  the  basis  of  risk  con¬ 
siderations  alone,  (it  should  be  noted  that  this  selection  is  not  based 


on,  considerations  of  national  utility;  political  pressures,  threat 


assessment,  etc;,  which  are  other  inputs  to  the  decision-maker V,  For 
example,  in  the  graph,  curves  /A  and  B  are  the  risk  profiles  of  cost, 
for  two  different  management  courses  of  action. 
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^Figure  12. 

Risk- :Prof lies  for-  Alternative  Courses  of  Action 

If  cQ  dollars  are  budgeted  jv  the  risk  is  the  same  for  each;  but  if  ' 
more  than  cQ  dollars  are  budgeted,  strategy  A  is  preferred..  Similarly, 
if  less  than  cq  dollars  are  budgeted  strategy  B  is  preferred., 

9.  A. discussion  of  the  major  assumptions  and  an  explanation  of  the 
disparity  when  the  results  are  different  from  those  expected.  This 
serves  not  oniy  as  a  check  on  the  work  of  the  risk  analysts  but  also 
enhances  the  credibility  of '  the  study  to  the  decision-maker. 

The  Use. and  Benefits  of  a  Risk  Analysis 

In  the  preceding  sections  of  this  chapter  we  have  addressed  the 
issues  df  "howr  to  perform  a  risk  analysis'"  and  what  the  outputs  of 
ah  analysis  should  be.  We  now  direct  our  attention  to  a  discussion 
of  how  a  risk  analysis  should  be  used  as  an  integral  part  of  the 
decision-making  process  and  the' benefits  which  accrue  from-,  its  use. 

A  risk  analysis  may  be  part  of  a  continuing  effort  by  %the  program 

>manager  to  keep  abreast  of  his  p3:ogram,  or  it  may  be  prepared  specif- 

\ 

ically  to  evaluate  alternatives  for  a  major  tradeoff  decision  by 
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higher  level  authority.  Either  case  cells  for  a  continuing  assessment 
qf  the  program  risks.  A  continuing  assessment  helps  to  determine  if 
there  is  a  decision  to  be  made,  while  a  risk- analysis  helps  determine; 
which  alternative  to  choose. 

There  are  at  least  three  significant  uses  for  a  risk  analysis. 

1 .  It  can  provide  the  Services  and., -more  specifically,  the 
Program:  Managg,  with  a  capability  to  identify  the  significant  sources 
of  ..program  risk' and  to  evaluate  eachof  the  individual  influences  to 
determine  the  proper  allocation  of  -  resources  and  efforts  for  the  reduce 
tion  and  control  of  ^these  ^risks ♦ 

The  identification  of  major  risks  and  uncertainties  as 
determined  by  the  assessment  should  be.  a  part  of  any  periodic  high  level 
review  for  major  programs.  In  the  past,  such  exposure  of  possible  pro¬ 
gram  problems  has  been  frowned  on  by  sesne  Program  Managers  for  fear  that 
the  problems  would  be  construed  to  be  the  result  of  poor  program  manage¬ 
ment.  It  ;is  time  that  ali  concerned  accept  the  fact  that,  in  complex 
weapon  system  development,  unexpected  problems  will  arise  due  to  the 
inevitable  lack  of  complete  knowledge  at  program  inception.  Furthermore, 
many  of  these  problems  are'  .outside  the  direct  control  of  the  Program 
Manager  and  all,  of  them  tend  to  intensify  if  left,  unattended..  For 
these  reasons  it  is  essential  that  emerging  problems  be  brought  to  the 
attention  of  a  high  level  review  as  soon  as  they  are.  identified. 

2.  Risk  Analyses  provide  the  military  Services  and  the  Office 
of  the  Secretary  of  Defense  with  necessary  information  bearing  on  the 
program  risks  ir.  order  to  select  an  appropriate  course-  of  action, 
establish  thresholds,,  or,  terminate  the  endeavor. 


The  Services  have  a  vital  need^for  a  thorough  risk  analysis 
of  all -alternative  systems  considered  to  meet  an  operational  requirement. 
In  many  cases  it  is  impossible  to  determine  today  just  hew  certain 
systems  evolved  as  the  answers  to  given  threats.  A  meaningful  and  well 
documented  analysis  of  the  risks  and  uncertainties  associated  with  each 
alternative  system  would  not  only  assist  the  Service  in  identifying  the 
best  program  to  present  for  negotiation  of  the  Development  Concept  Paper, 
but  would  tend  to  keep  the  Service  in  a  defensible,  position  on  that  pro¬ 
gram  throughout  the  Conceptual  Phase.  The  review  of  the  preferred 
alternative  by  the  DSABC  (or  Congress)  at  each  major  milestone  calls  for 
and  vill  require  a  formal  risk  analysis.  In  the  charter  far  the  DSABC, 

authority  has.  been  granted  to  establish  working  groups  to  assist  the 

/ 

Council  members  in  their  reviews.  One  such  working  group,  should  be 
assembled  to  consider  the  analyses  presented  by  the  Services  at  the  time 
of  proposed  entry  of  a  major  program  into  validation  or  into  full  Scale 
Development.  The  recommendation  of  this  group  should  be  one  of  the 
inputs?  to  the  DSARC  decision  at  that  milestone. 

3.  Risk  Analysis  provide  the  Program^  Manager -with  more  complete 
information  on  which  to  base  the  Initial  Cost  Estimate  ( ICE), and  a  more 
meaningful  method  of  presenting  the  Estimate  to, the  DSARC  and  Congress. 

The  Program  Office  is  normally  established  at  the  time  the 
operational  requirement  is  defined;  It  is  anticipated  that  the  early 
risk  analysis  performed  to;  select  the-  best  system  during  .Concept' 
Formulation  would  form  the  basis  for  the  Program  Director  to  develop1 
a  comprehensive  list  of  technical  uncertainties  to  be  addressed  by  the 
contractors  in  competition  for  the  development  contract.  Since,  the 
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Program  Manager  iff  normally  responsible  for  preparing  the  cost  estimate- 
to  present  to  the  DSARC  at  the  conclusion  of  the  Concept  Formulation 
Phase  and  for  use  as  the  Selected  Acquisition  Report  (SAR)  baseline,  it 
behooves  hi*  to  coordinate  closely  with  the  Source  Selection  Authority 
to  insure,  that  the  contractors  analyze  and  document  all  the  major  problems 
in  developing  the  .specified  system  and  include  the  possible  effects  of 
such  problems  in  their  cost  estimates.  From  7x>th  the  Service's  and  the 
DOD's  viewpoints,  it  is:  imperative  that  the  most  realistic  ICE  be 
reported  in:  tb«s  SA%  This  can  be  accomplished  only  as  a  result  of  a 
thorough  risk  analysis.  Ho  one  entity  affects  apparent  cost  growth 
more  than- the  Initial  Cost  -Estimate. 

During/the  cooling-off  period,  between  source  selection  and  contract 
awards  the  winning  contractor' brisk  analysis  should  be  updated  to 
reflect  any  changes  that  may  ;have  been  oade  in  the  stated  operational 
requirements.  The  Pregram  Manager  should  then  use  that  risk  analysis 
to  present,  for  the  current  required  performance ,  a  joint  risk  profile 
of  time  and  cost  to  the  DSARC.  (Of  course,  the  Prograd  Manager  will 
have  already  determined -a  desired  budget  and  schedule  frvn  the  profile.) 
The  profile  would* illustrate  feasible  ranges  of  cost,  and  schedule j  and 
.aid  the  Program  Manager  in  immediate,  negotiation  of  his  position,  if 
that  becomes  necessary.  From  such  a  risk  profile,  -the  DSARC  could  also 
determine  the  program  risk  associated  with  the  thresholds:  stated  earlier 
in  the  DCP. 

The  primary  benefits  of  a  formal  risk  analysis  derive  from  an 
increased  degree,  of  realism  and  thoroughness,  which  this  analysis 
injects  into  the  program.  It  is  the  means  by  which  the  manager  choses 
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fro*  among  the  alternatives  available  to  himin  a  deci  si  on- waking 
environment  characterized  by  over-optimism,  uncertainty  and  change.  As 
many  members  of  government  and  industry  pointed  out  to  us  during  the 
preparation  of  this  report,  "A  formal  risk  analysis  is  putting  on  the 
table  thC3e  problems  and  fears  which  heretofore  mere  recognized  but 
intentionally  hidden."  What  follows  is  a  summary  of  the  major  benefits' 
of  a  formal  risk  analysis . 

1.  A'  gcod  risk  analysis  at  the  start  of  a  program  is  one  of  the 
best  ways  to  aid  management  in  carrying  out  a  careful,  detailed  program 
planning  operation,  and  to  better  fix  the  constraints  of  a.  program 
(cost,  schedule,  performance)  at  the  earliest  phases  where  the  payoff 

is  greatest.  Indeed,  risk  itself  cannot  be  defined  until  the  constraints 
themselves  are  clearly  defined.  The  'fixing  of  constraints  thus  becooes 
an  iterative  process  which  involves  examining  the  risk  of  adhering  to 
the  candidate  constraints. 

2.  By  focusing  proper  attention  on  eaph  and  every  program 

activity  and  event,  the,  likelihood  of  adverse  surprises  is'  -  . . 

greatly  reduced.  To  say  the  least,  a  good  risk  analysis  constitutes 
an  extremely  extensive  planning;  effort  in  which  the  aim  is  the  identi-. 
fication  of  all  expected  rough  'spots  in  the  entire  program.  Change, 
when  it  comes,  is  anticipated  iind  the  alternatives  available  for  dealing 
.with  those  problems  which  induce,  change  are  planned  in  advance,  thus 
reducing  undesirable  impacts  on  program.activities . 

3’.  The  careful  analysis  and  planning  operation  rioted  above 
will  assure  a  total  systems  approach  in  which  engineering  functions, 
program  budgets,,  and  time  schedules  are  viewed  together.  This  resuits 
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in  .balanced  program  planning,  each  facet  of  ,the  program  receiving  its 
fair  attention  and  evaluation,,  and  enables  program  personnel  to  better 
.estimate  the  ability  of  the  program  to  reamin  within  the  established 
constraints . 

4.  If  a  risk  analysis  is  performed  at  the  start  of  a  program 
and  if.  it  is  continuously  updated,  the  problem  areas  will  be  identified 
as  early  as  possible..  This  will  permit  the  earliest  and  the  most  timely 
scheduling  of  appropriate  efforts  aimed  at  their  solutions; 

5.  With  marl  mum  advance  notice  of  the  occurrence  of  specific 
problems,;  program  management  will  be  better  able  to  optimize  the 
allocation  of  funds  to  the  various  program  activities.  This  enhances 
the  likelihood  of  adequate  apportionment  of  personnel  and  facilities 
geared  to  a  realistic  assignment  of  priorities,. 

6.  The  very  process  of  performing  a  complete  risk  analysis 
forces  the  collection  of  much  useful  information  with  attention  focused 
on  critical  information  gaps. 

7.  One  of  .the  major  benefits  of  risk- 1  analysis  may  be  that  it 
alerts  management  to  the  need  for  replacing  unrealistic  "single  value" 
estimates  by  probability  distributions.. 

8.  As  the  .accuracy  of  management  and  design  approaches  is 
increased  through  the  use  of  probabilistic  parameter  descriptions,  it 
is  expected  that  there  will  be  a  better  chance  of,  -obtaining  an  optimum 
system  configuration. 

9.  A  good  analysis  of  risk  is  basic  to  the  argument  needed  to 
justify  budget  requests.  Officials  are  more  likely  to  be)  convinced  by 
sound  analysis  than  by  some  rather-  general  statement  of  belief— they 
want  credibility. 
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'10-;-  Sisk  analysis  provides  a  management  tool  for  evaluating  the 
decisions  of  the  Program  Manager. 

Risk  Analysis  in  the  Organization  Structure 

The  concept  that,  a  risk  analysis  requires  an  iterative  interaction 
between  the  manager  and  the  assessors,,  coupied  with  the  fact  that  the 
contractor  has  the  data,  dictates  that  the  analysis  must,  be  performed 
in  either  the  military -or  contractor  Program  Office.  Since  the  military 
and  contractor  managers  for  major  programs^  are  in  almost  daily  contact, 
the  most  efficient  scheme  would  be  to  have  the  assessment  done  by  the 
contractor,  with  the ■ selection  of  alternative,  courses  of  action  made 
by  the  , Military  Program  Manager  on  recommendations,  from  the  Contractor 
Program  Manager.  This  arrangement  keep,?  the  military  FM— the  one  with 
most  to  gain  from  a  risk  ■,ana^-ais‘i-ia  constant  .touch  with  those  factors 
affecting  his  program. 

The  picture  that  has  been  painted  thuk  far  is  primarily  one  of 
program  advocacy— as.  it  should  he  if  the  Program  Manager  is  to  be 
expected  to  include  such  an  analysis  in  his  management  techniques  with 
any  degree  of  utility  and  enthusiasm.  However,  if  the  Program  Manager  is 
faced  with  a  decision  that  requires  tradeoffs  outside  the  -thresholds  of 
his  charter,  he  should  he  prepared  to  present  a  complete  risk  analysis 
with  several  alternative  courses  of  action  and  his  recommendations  to 
a  high  level  decision  board  within  his  Service  having  major  tradeoff 

authority.  At  the  same  time,  PM's  of  major  programs  should  be  pro- 

,  * 

vlded  direct  access  to  such  a  board.  This  hoard,  or  review -council, 
in  turn,  which  may  vary  from  Service  to  Service,  /should  fund  an 
independent  evaluation  of  the  risk  analysis  to  aid  in  making  tradeoff 
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decisions.  This  evaluation  could  be  dotfe-by  another  contactor,  a 
non-profit  organization,  a  consulting  firm,  ox*  a  Service  organization, 
and  would ^provide  a  check  on  the  accuracy  and  completeness  of  the 
analysis  submitted  by  the  Program  Manager's  analysis.  The  results  of 
the  evaluation  should  be  presented  simultaneously  to  the  Program  Manager 
and  the  Council 's  staff  and  included  in  the  subsequent  tradeoff  dis¬ 
cussions.  Furthermore,  for  major,  tradeoffs  both  the  military  and  the 
contractor  ^Program  Managers  should  be  present  to  interpret,  the  content 
and  sourch  of  the: analysis. 

The  Service  Council  or  DSABC  risk  analysis  staff  should  be  minimal 
in  number,  and  serve  as  the  administrative  and  advisory  link  between 
the.  Council  and  the  .independent  evaluation  group.  (Note  that  we've  used' 
the  word  "evaluation;’,"  and-  not  "analysis .”)  No  independent  group  will 
ever  be  able  to  effectively  perform  a  risk  analysis  on  a  program. 

Because  they  are  independent  they  will  neither  have  access  to  the  vast 
amount  of  data  necessary;  to  do  the  assessments,  nor  will  they  be  able 
to  efficiently  interact "with  the  Program  Manager  to  obtain  the  iterative 
selection  of -alternative  courses  of  action  to  assess. 
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FIGURE  14. 

RISK  ANALYSISORGANIZATIONAL 
STRUCTURE  WITHIN  DOD 


Figure  14  shows  a  proposed  risk  analysis  organizational  .structure 
within  DOD.-  Note  that  only  the  preferred  course'  of  action  is  pre¬ 
sented  to  the  DSARC.  The  innovative  feature  of  this  structure  is 
that  risk  profiles  of  time  and  cost,  as  opposed  to  point  estimates, 
are  presented  for  review. 
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CHAPTER. VI:  CONCLUSIONS  AND  RECOMffiNDATIONS 

The  goal  of  this  study  has-been  to  examine  the  DOD  Weapons  Systems 

/ 

Acquisition;  Process  to  see  if  the  concept  of  Risk  Analysis  can  contri¬ 
bute..  We  conclude  that  techniques  exists  for  conducting  a. risk  analysis, 
and  that  such  an  analysis  can.be  a  valuable  management  tool.  Guidance 
has  been  provided  for  its  preparation -.'and  use. 

The  conclusions  which  follow  have  emerged  as  salient  points  of 
our  study.  Each  conclusion  is  accompanied  by  a  recommendation  for  action 
that  we  feel  will  help  establish  the  pole  of  risk  assessment  and  analysis 
in  the  acquisition  cycle.  We.  by  no  means  consider  this  an  exhaustive 
list  of  proposals  for  change  to  present  .acquisition  methods.  The 
recommendations i given  here  are  intended  to  augment  the  suggestions  giyen 
in  the  several  other  recent  documents  on  weapon  system  acquisition. 
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CONCLUSION  1: 


ONE  OF  THE  BASIC  PROBLEM  IN  ANY  STUDY  ON  RISK  IS  THE  LACK  OF' A 
GENERALLY  ACCEPTED  GROUP  OF  DEFINITIONS.  "Risk"  is  often  used  to  mean 
the  “probability  of  failure,"  but  some  studies  include  the  concept  of 
the  "impact  of  failure" .in  their  definition  of  risk.  Also,  many  manage¬ 
ment  officials  have  used  the  term  "risk"  in  a  context  where  "uncertainty" 
would  be  more  appropriate.  Thus,  the  general  "uncertainties”  of  the 
weapons  acquisition  process  become  confused  with  the  more  specific 
"technical  risks.1! 


RECOMMENDATION - 1: 

ESTABLISH  DOD  DEFINITIONS  OF  THE.  BASIC' TERMS  AND  CONCEPTS  USED  IN 
RISK  ANALYSIS .  A  list  of  candidate-definitions  is  as  follows; 

RISK:  The  probability  that  a  planned  event  will  not  be' 
attained  within  constraints  (cost,  schedule, 
performance)  by  following  a  specified  course  of 
action. 

;  ^  M  * 

UNCERTAINTY :  Incomplete  knowledge . 

RISK  ASSESSMENT :  A  comprehensive  and  structured  process- 

for  estimating  the  risk  associated  with  a 
particular^  alternative  course  of  action; 
also  the  product  of  such  a  process. 

RISK ^MANAGEMENT :  The  generation  of  alternative  courses 
of  action  for  reducing  risk. 

RISK  ANALYSIS:  The  process  of  combining  the  risk  assessment 
with  risk  management  in  an  iterative  cycle; 
also  the  product  of  such  a  process . 


CONCLUSION  2: 

TECHNICAL  UNCERTAINTY  MAY  BE  ONLY  A  SMALL  BART  CF  THE  WEAPON  SYSTEM 
ACQUISITION  PROBLEM.  SUCCESSFUL  PROGRAM  DIRECTION  ALSO  REQUIRES  MANAGE¬ 
MENT  OF  TARGET,  INTERNAL  PROGRAM,  AND  PROCESS  UNCERTAINTIES. 

RECOMMENDATION  2: 

REQUIRE  ANY  "RISK  ANALYSIS"  TO  INCLUDE  CONSIDERATION  OF  TARGET, 
TECHNICAL,  INTERNAL  PROGRAM,  AND  PROCESS  UNCERTAINTIES., 


CONCLUSION  3: 

IN  THE  AREA  OF. QUANTITATIVE  RISKASSESSMENT,  AGGREGATION  TECHNIQUES 
(SUCH  AS  NETWORK  ANALYSIS)  ARE  .FAR  MORE  ADVANCED  THAN-  THE  TECHNIQUES 
FOR  OBTAINING  INPUT  DATA  (SUCH, AS  SUBJECTIVE  PROBABILITY  AND 
TECHNOLOGICAL  FORECASTING j .  . 

RECOMENDATION  3: 

'  I 

FUNDING  PRIORITY  FOR  IMPROVING  METHODS  FOR  QUANTITATIVE  RISK 
ASSESSMENT' SHOULD  BE  - GIVEN  TO  DEVELOPMENT  OF  INPUT  TECHNIQUES  . 
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CONCLUSION  4: 

RISK  ANALYSIS  PRIMARILY  BENEFITS  THE  PROGRAM’ MUIAGER ..  NO  AGENCY 
OUTSIDE  TEE  MILITARY  OR  CONTRACTOR  PROGRAM' .OFFICE  CAN  EFFECTIVELY  PER¬ 
FORMS  RISK  ANALYSIS  OF  THAT  PROGRAM.  Only  the  MiUtW  Program  Office 
and  the  Contractor's  Program  Office-  have  access  to  the  vast  amount  of 
data  necessary  for  the  assessment  and  effective  contact  with  the  Program 
Manager  for  selection  of  alternative  courses  of  action.  Such|a  risk 
analysis  valid  add  a  new  dimension  to  the  presentation  of  alternative 
courses  of  action  to  the  level  of  management  entrusted  vith  major  trade¬ 
off  authority;,  particularly  vhen  supplemented  by  an  independent  evaluation 
of  the  risk  analysis  itself. 

RIiCO!#lENDATIONU:, 

— — .. 11  :■■■„ - >> 

o  -Direct  each  concept  formulation:  contractor,  .to  perform  a,  -risk, 
analysis  of  the  system  being  proposed  for  negotiation  of  the  Development 
Concept  Paper; 

o  Direct  each  contractor,  in  the  source  selection  competition  to 
perform  a  risk  analysis  and  specifically  include  at  least  the  uncer¬ 
tainties  identified  hy  \the  risk  analysis;  performed  during  concept 
formulation. 

o  Require  che  'Program  Manager  and  the  y inning  contractor  to  update 
the'  risk  analysis  after  source  selection  and  before  contract  award  if 
adjustments -are  made  to  the,  stated  operational  requirements  during 
that  period. 

b  Direct  the.  contractor  for  each  major  on-going  program  to  conduct 
a  continuing  assessment  of  the  risk  in, the  program.  The  Program 
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RECOUgglATIOK  U  (Contd): 

*  O 

Manager  should-  use  the  results  of  the  continuing  assessment  in  his  day 
to  day  management^ decisions. 

o  Require  the  Program  Manager  to  present  the  results  of  a  current 
risk  analysis,  to  include  alternative  courses  of  action,  each  time  the 
program  is  reviewed  by  higher  service  authority  for  a  major  tradeoff 
decision. 

o  Require  the  Program  Manager  to  present  the  results  of  a  risk 
assessment  of  his  program,  specifically  the  risk  profiles  for  time  and 
cost,  each,  time  the  program  is  reviewed  by  the  DSARC  or  Congress. 

o  Require  an  independent  evaluation  of  the  risk  analysis  or 
assessment  each  time-  the  program  status  is  presented  to  higher 
authority  for  a  major  tradeoff  or  milestone  review n 
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CONCLUSION)  5:- 

INITIAL  COST  AND  SCHEDULE  ESTIMATES  FORMAJOR  PROGRAMS  HAVE 
INVARIABLY ’.BEEN  OVER-OPTIMISTIC.  THE  RISK  THAT  COST  AND  SCHEDULE 
CONSTRAINTS  WILL  NOT  BE  MET  CANNOT  BE  DETERMINED  TF-COST  AND  SCHEDULE 
ESTIMATES  ARE  GIVEN  IN  TERMS  OF  SINGLE  POINTS  RATHER  THAN  DISTRIBUTIONS . 

RECOMMENDATION  5: 

REPLACE  THE  POINT  ESTIMATES  OF  COST  AND  SCHEDULE. PRESENTLY  USED 
WITH  THE  JOINT  RISK  PROFILE  FOR  COST  AND  SCHEDULE  FROM  THE  RISK  ANALYSIS 
DEVELOPED  AT  THE  COMPLETION’  OF  SOURCE  SELECTION.  At  the  present  time, 
this  can  best;  be  obtained  front  the  output  of  a  versatile,  simulated 
network  approach  using  inputs  from  gproup  assessment  techniques, 
subjective  probability,  technological  forecasting,  cost  estimating, 
and  others  as  sources. 


CONCLUSION  6:. 

TO  OUR  KNOWLEDGE  NO  MAJOR  BOD  PROGRAM  HAS  DEVELOPED  OR  USED  A  RISK 
ANALYSIS  OF  THE- MAGNITUDE  ENVISIONED  IN  THIS  REPORT. 

p  5  ‘  -  ■**  ’  ^  ' 


RECCMCBDATION  6: 

'■■■  ■  1  1  ;  ■»  % 

INITIATE  TEST  CASES  BNEJIATELY*  FORMAL  RISK, ASSESSMENTS  AND  ANALYSES 
SHOULD  BE  USED  THROUGHOUT  THESE  PILOT  PROGRAMS  TO  DETERMINE  THEIR  FEASI- 

o  A  w  *  **■  ‘ 

BILITY  AND  UTILITY  TO  A  DECISION  MAKER. 

For  a  thorough  trial,  prototype  risk  analysis  programs  should  be 

initiated  in  programs  in  each  of  the  three  phases  beloyr: 

o  Concept  Formulation 
o  Validation 
6  Full  Scale,  Development 

The  characteristics  of  a  Risk  Analysis  prototype  should  include: 

o  A  major  program  .or  subsystem 
o  A  relatively  short  duration 
o  Comparison  with  a  regular  program 
o  A  formal  risk  assessment  done  by  the  contractor 
o  An  evaluation  of  the  contractor’s  risk  analysis  by  ah 
independent  agency  (another  contractor,  non-profit 
corporation,  consulting  firm,  etc.) 

The  pilot  program  should  assist  in  evaluating, and  determining: 

o  Procedures  for  risk  assessment 
o  Appropriate  team. composition 
6  Input  data  requirements  for  ,a  risk  assessment 
o  Methods  of  data  presentations  to  a  decision  maker 
o  Outputs  of' an  acceptable  Risk  Analysis, 

6  Othdr  problem  areas . 


CONCLUSION.  7: 

THERE  IS  NO  "ONE  BEST  WAY"  TO  CONDUCT  A  BISK:  ANALYSIS .  There  are 
many  quantitative  and  qualitative  methodologies  available  to  assist 
the  analyst.  The  procedures'  to  be  followed  for  a;  given  program  should 
be  left  to  the  discretion  of  the  Program  Manager  with  the  advice  of  the 
analysis  team.  The  common  denominator  that  should  be  expected  is 
uniform?  output  information. 

RECOHiffiNDATICKf  7 :  g  / 

THE  FOLLOWING  LIST  OF  OUTPUTS  F OR  A  RISK  ANALYSIS  IS  RECOMMENDED. 
PILOT  PROGRAMS  SHOULD  RECOMMEND  THE  MOST  USEFUL  OUTPUTS'. 

1.  A  general  description  of  the  types  of  uncertainties  in  the 

program. 

2 .  A  detailed  discussion  of  the  potential  problems  in  each 

major,  program  element  (engine,  etc.). 

/ 

3.  Identif cation  of  alternate  management  courses  of  .action  to 
resolve  the  major,  uncertainties. 

4.  Probability  distributions  \of  time  and.  cost;  risk  profiles 
for  each  course  of  action. 

5 .  A  sensitivity  analysis’  to  determine  the  effect  of  input 
perturbations  on  the  risk  assessment  output. 

6.  Tradeoff  studies  as  directed  by  the  Program  Manager  for 
maintaining  the  overall  program  within  specified  cost,,  time  and 
performance  thresholds. 

7»  Comparison  with  previous  risk  analyses  to  identify  trends. 


RECOHCNDATICW  -?  (Contd): 

8.  A  -  coiqparison  of.  the.  candidate  courses  of  action  and  a 
recommendation  of  apref  erred  strategy;  based  on'  risk  considerations 
alone . 


9.  A  discussion  of  the  major  assumptionsandan  explanation 
of  the  disparity  when  the  results  are  different  from  those  expected. 


CONCLUSION  8: 

THE  CONCEPTS  OF  RISK  ANALYSIS  ABE  INADEQUATELY  UNDERSTOOD.  AN 
EDUCATION  PROGRAM  IS  NEEDED  TO  INSTRUCT  ANALYSTS  AND  MANAGERS  INTHE' 
PREPARATION  AND  USE  OF  A  FORMAL  RISK  ANALYSIS. 

At  the,  present  time-,  there  are  very  fey  people  in  vthe  military 

i 

services  of  industry  who  sere  qualified  by  experience  to  perform  a 
quantitative  jrisk  analysis.  Similarly,,  there  are  fev  managers  who  are 
accustomed  to  using  the  outputs  of  a  risk  analysis.  For  instance, 
probability  distributions  more  accurately  depict  the  risk  of  development 
than  do  point  estimates;  yet  there  is1  widespread  resistance  to  using  a 
probability  distribution  because  it  is  an  unfamiliar  or  perhaps  suspect 
technique. 

RECOMMENDATION  8; 

FOLLOWING  A  MEANINGFUL  PILOT.  PROGRAM,  DESIGN  AND  IMPLEMENT  A  RISK 
ANALYSIS  EDUCATION  PROGRAM  WITH  THE  FOLLOWING  MAJOR  AREAS  OF  EMPHASIS; 

Short  orientation-,  courses  in  risk  analysis  for  high-level 
officials  who  deal  with  uncertainties  in  program  management  and  program 
approval. 

Longer  training  courses  to  outline'  the  details  of,  risk  assess¬ 
ment  techniques  for  selected,  personnel  who  may  be  required  to  perform 
such  assessments,  in  the  government  and  industry  Program  Offices. 

Introduction  of  Risk  Analysis  as  a.  discrete  subject  in  the 

curriculum  of  appropriate  government  schools  such  as  the  Defense  Weapons 

-  <. 

System  Management  Center. 
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BECOl^ffilCDATION  8  (Contd)-:, 

Assignment  oiL specially  talented  military  and, civil  service 
personnel  to  a’  non-profit  _ corporation  or  private  consulting  firm  with 
expertise  in  Risk.Analysfs.  Such  selectees  should  already  -have  a 
Master's  degree  in  operations  research,  systems  analysis,  or  some 
related  discipline  and,  their  assignment  (for  up  to  a  year)  would 
qualify  them  to  conduct  formal  Risk  Analyses  for  the  Services.  This 
on-the-job  training  would!, significantly  increase  the  Service  capa¬ 
bility  to  perform  and  use  Risk  Analyses  and  has  its  precedent  in  the 
"training  with  industry"  program. 
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